ROMS-JEDI Compiling

From WikiROMS
Jump to navigationJump to search
Compiling ROMS-JEDI
ROMS-JEDI Menu
  1. Introduction  
  2. Compiling  
  3. Implementation  
  4. Observations  
  5. Unit Test Cases  


Configuring GIT

Before downloading the ROMS-JEDI repository, ensure that your ~/.gitconfig has the appropriate git-lfs configuration for correctly downloading the default application (U.S. West Coast, WC13 option) input and observation NetCDF files. Otherwise, the Unit Test Cases requiring input NetCDF files will fail. The Git LFS is a command line extension and specification for managing large files with Git. A sample of the configuration file looks like this:

more ~/.gitconf

[User]
name = GivenName MiddleName FamilyName
email = your@email
[credential]
helper = cache --timeout=7200
helper = store --file ~/.my-credentials
[filter "lfs"]
clean = git-lfs clean -- %f
smudge = git-lfs smudge -- %f
process = git-lfs filter-process
required = true

Alternatively, you may execute git lfs pull at your location of this repository to download a viable version of the Git LFS files from the remote repository in GitHub. Also, to automatically add the LFS filter to your existing ~/.gitconfig, you could use git lfs install from anywhere in your computer directories.

Downloading ROMS-JEDI

The ROMS-JEDI repository is private within the Joint Center for Satellite Data Assimilation (JCSDA) community. Users need to be partners, contributors, or members of JCSDA to access all the JEDI components that are part of the abstract layer infrastructure centered around the Object Oriented Prediction System (OOPS).

Use the following command to download the ROMS-JEDI interface:

git clone https://github.com/JCSDA-internal/roms-jedi.git

Requirements

  • Make sure you have the spack-stack software installed for either gfortran or ifort containing several packages need to run the ROMS-JEDI interface. For more information, please check https://github.com/JCSDA/spack-stack.

    In our computers at Rutgers University, the spack-stack can be loaded by executing either:
    module purge; module load stack-gcc; module list (gfortran compiler)
    module purge; module load stack-intel; module list (ifort compiler)
    Installing the spack-stack is not trivial and requires computer software expertise.
  • We also like to define the MPIRUN environmental variable to specify the MPI executable in a particular computer (say, srun, mpirun, etc.). We need the full path for ctest to work. Notice that we specify this executable in the ecbuild command below.

    For example:
    setenv MPIRUN /opt/slurm/bin/srun
  • Several environment variables can be activated for verbose debugging of the building blocks for the ROMS-JEDI interface: (use 1 or true to activate and 0 or false to deactivate)
    setenv LdebugGeometry 1
    setenv LdebugAnalyticInit 1
    setenv LdebugField 1
    setenv LdebugFields 1
    setenv LdebugFieldsUtils 1
    setenv LdebugGeometry 1
    setenv LdebugLinearModel 1
    setenv LdebugLinearModel2Geovals 1
    setenv LdebugModel 1
    setenv LdebugModel2Geovals 1
    setenv LdebugTrajectory 1

Compiling ROMS-JEDI Default Application WC13

By default, ROMS-JEDI is configured with a coarse (54x53x30 grid points) US West Coast (WC13) application. See 4DVar Tutorial Introduction for detailed information about the WC13 configuration. However, any other ROMS application can be configured by including in the ecbuild command the directives -DROMS_APP=MyAppCPP and -DROMS_APP_DIR=MyAppDirPath.

The strategy is to create a Bundle and build subdirectory for each ROMS-JEDI application. For WC13 (default), we create Bundle_wc13 and build_wc13 below. Please follow the following steps to configure ROMS-JEDI in your computer:

git clone https://github.com/JCSDA-internal/roms-jedi.git  !> If the first time on your computer

cd roms-jedi  !> ROMS-JEDI interface directory, <interface_dir>
mkdir Bundle_wc13
cp bundle/.gitignore bundle/CMakeLists.txt Bundle_wc13
mkdir build_wc13
cd build_wc13

ecbuild -DMPIEXEC_EXECUTABLE=$MPIRUN -DMPIEXEC_NUMPROC_FLAG="-n" -DCMAKE_BUILD_TYPE=Release ../Bundle_wc13
make -j 10

cd roms-jedi  !> sub-directory: <interface_dir>/build_wc13/roms-jedi
ctest -N  !> lists all the Unit Tests available
ctest -E -V get_  !> runs all the Unit Tests

cd test/Data  !> sub-directory: <interface_dir>/build_wc13/roms-jedi/test/Data
 !> to check the results in various sub-directories

The ecbuild command will clone the required JEDI components that are specified in Bundle_wc13/CMakeList.txt. For example:

# Required repositories

ecbuild_bundle( PROJECT roms GIT "https://github.com/myroms/roms.git" BRANCH develop UPDATE )
ecbuild_bundle( PROJECT oops GIT "https://github.com/jcsda-internal/oops.git" BRANCH develop UPDATE )
ecbuild_bundle( PROJECT vader GIT "https://github.com/jcsda-internal/vader.git" BRANCH develop UPDATE )
ecbuild_bundle( PROJECT saber GIT "https://github.com/jcsda-internal/saber.git" BRANCH develop UPDATE )

ecbuild_bundle( PROJECT gsw GIT "https://github.com/jcsda-internal/GSW-Fortran.git" BRANCH develop UPDATE )
ecbuild_bundle( PROJECT ioda GIT "https://github.com/jcsda-internal/ioda.git" BRANCH develop UPDATE )
ecbuild_bundle( PROJECT ufo GIT "https://github.com/jcsda-internal/ufo.git" BRANCH develop UPDATE )

ecbuild_bundle( PROJECT roms-jedi SOURCE ../ )

The ecbuild tool builds by default the RelWithDebInfo (-O2 -g options) for optimized with debugging information version of JEDI and ROMS-JEDI, rendering slower execution. However, for faster and optimized execution, we include the directive -DCMAKE_BUILD_TYPE=Release to the ecbuild command. For debugging with TotalView, we include instead the directive -DCMAKE_BUILD_TYPE=Debug, which is much slower.

Compiling other ROMS-JEDI Applications

Any ROMS application can be run in ROMS-JEDI by specifying the appropriate CPP option and necessary input files. Only WC13 is provided with the interface. However, we facilitate how to run such applications. Notice that the WC13 configuration and input files are located in roms-jedi/test/Applications/wc13. The template input YAML files are located roms-jedi/test/templates. We use a Perl script to create the application input YAML files from the templates. Please check in roms-jedi/test/Applications/wc13/testinput.

For example, if running the U.S. East Coast DOPPIO application (240x104x40 grid points) or any other, please follow the following steps to configure ROMS-JEDI:

cd roms-jedi  !> ROMS-JEDI interface directory, <interface_dir>
mkdir Bundle_doppio
cp bundle/.gitignore bundle/CMakeLists.txt Bundle_doppio
mkdir build_doppio
cd build_doppio

ecbuild -DMPIEXEC_EXECUTABLE=$MPIRUN -DMPIEXEC_NUMPROC_FLAG="-n" -DROMS_APP="DOPPIO" \
-DROMS_APP_DIR="DOPPIOpath" -DCMAKE_BUILD_TYPE=Release ../Bundle_doppio


make -j 10

cd roms-jedi  !> sub-directory: <interface_dir>/build_doppio/roms-jedi
ctest -N  !> lists all the Unit Tests available
ctest -E -V get_  !> runs all the Unit Test
cd test/Data  !> sub-directory: <interface_dir>/build_doppio/roms-doppio/test/Data
 !> to check the results in various sub-directories