VERY IMPORTANT: Model Coupling with the ESMF/NUOPC Library

[arango]

I have been developing the coupling with the Earth System Modeling Framework (ESMF) library with the National Unified Operational Prediction Capability (NUOPC) layer for a while. It was challenging to code the coupling infrastructure from scratch. It took lots of debugging. For more information about this development, check ​WikiROMS. We will continue expanding the online documentation.

Currently, the ROMS coupling ESMF/NUOPC interface can be illustrated with the following diagram.

It includes four distinct compartments: Driver, Model Components, Coupler, and Connectors. Righ now, we are considering five classes of coupled Earth System Model (ESM) components: ROMS, ATM, DATA, SEA-ICE, and WAVE models. There is a Fortran module that sits on of each ESM component and referred as the NUOPC cap file. It is a Fortran code layer making calls to the ESM numerical kernel via the initialize, run, and finalize phases

There is a separate cap file for each coupled ESM component, which, from the ROMS perspective, corresponds to the atmosphere, sea-ice, wave, data models. Therefore, it is an abstract block that allows ROMS to communicate and exchange data seamlessly within the ESMF/NUOPC framework. The ROMS and other ESM components grids may (and usually do) have a different geographical extent and horizontal resolution. So if, for example, the atmosphere component domain is larger than the ROMS domain, it is necessary to provide data from another source to the atmosphere model at all surface grid points outside the ROMS domain. Therefore, a DATA component and its cap file are also required for most applications. In this case, SST is exported from ROMS and DATA to the atmosphere component, which imports the regridded values and then melds both SST fields. The melding is computed as:

SST^ATM(:, :) = W^ROMS(:, :) * SST^ROMS(:, :) + W^DATA(:, :) * SST^DATA(:, :)

where W^ROMS(:, :)+ W^DATA(:, :) = 1 are the weight coefficients that warantee a smooth transition between the ROMS and DATA values, as illustrated below for the Hurricane Irene application.

These weights are computed with the script coupling/wrf_weights.m of the ROMS Matlab repository.

The design of the ROMS coupling interface with the ESMF/NUOPC library allows both driver and component modes of operation:

In the driver mode, esmf_driver.h is the main program that provides all the interfaces and logistics to couple to other ESM components. In addition, it provides NUOPC-based generic ESM component services, interaction between gridded components in terms of NUOPC cap files, connectors between components for the regridding of source and destination fields (esmf_coupler.h), input scrips, and coupling metadata management.

In the component mode, the esmf_roms.F cap module is provided. It can be adapted and incorporated into other NUOPC-based coupling systems, like the NOAA Environmental Modeling System (NEMS) or the Community Mediator for Earth Prediction Systems (CMEPS).

All the ESMF/NUOCP source codes are located in the Master sub-directory:

• mod_esmf_esm.F: It defines all the structures and variables for ESM coupling. It also includes several support routines. It is fully tested.

• esmf_driver.h: Main program for the driver mode of operation. It initializes ESM coupling, creates ESM components, process ESM configuration, register ESM components, initialize ESM components, run ESM components, finalize ESM components, and finalize ESMF coupling. It is fully tested.

• esmf_coupler.h: Computes, executes, and releases RouteHandle connectors used for the regridding of source and destination fields. It is fully tested.

• esmf_esm.F: Sets NUOPC generic methods, entry points for each coupled components connectors, sets system internal clock, and sets coupling RunSequence. It is fully tested.

• esmf_roms.F: ROMS cap file. Sets NUOPC generic methods for initialize, run, and finalize. Checks import fields, sets grid component object, sets import and export states, advances ROMS for a couple interval, and sets import and export routines for the exchange of fields with other ESM components. It is fully tested.

• esmf_data.F: DATA cap file. It is used to provide data to ESM components uncoupled fields or to provide values at locations not available because of noncoincident grids. The data comes from specified input files, usually NetCDF files. It is fully tested.

• esmf_atm.F: Generic module for the atmosphere model component.
  
  
  • esmf_atm_coamps.h: Coupled Ocean-Atmosphere Mesoscale Prediction System (COAMPS, Hodur, 1997; Chen et al., 2014) cap file. It is fully tested in applications for the US West Coast and Indian Ocean. It can be used for both free and restricted NRL versions.
    
    
  • esmf_atm_regcm.h: ICTP Regional Climate Model (RegCM, version 4.5 and up, Elguindi et al., 2014 ) cap file. RegCM is maintained by the International Centre for Theoretical Physics (ICTP), Trieste, Italy. It was built upon the NCAR-PSU Mesoscale Model version 4 (MM4; Dickinson et al., 1989; Giorgi, 1989). This cap file is based on RegESM (Turuncoglu and Sannino, 2016).
    
    
  • esmf_atm_wrf.h: Weather Research and Forecasting (WRF, Skamarock et al., 2005) model cap file. It is fully tested in regional applications for the US West and East Coasts. WRF is maintained by a consortium of US agencies and institutions (NCAR, NCEP, FSL, AFWA, NRL, FAA, and University of Oklahoma).

• esmf_ice.F: Generic module for the seaice model component.
  
  
  • esmf_ice_cice.h: Los Alamos Sea Ice (CICE) model cap file. It is not fully tested. We need experts with applications to help us test this coupling component.

• esmf_wav.F: Generic module for the wave model component.
  
  
  • esmf_wav_wam.h: ECMWF Wave Model (WAM, WAMDI Group, 1998) cap file. This cap file is adapted from RegESM (Turuncoglu and Sannino, 2016).
    
    
  • esmf_wav_ww3.h: Wavewatch III Model (WW3, WW3DG 2019) cap file. It is not uploaded in this release, but it will be available in the future.
    
    

The ROMS cap file has the following routines:

MODULE esmf_roms_mod

  ROMS_SetServices       Sets entry points using NUPOC generic methods for initialize, run, and finalize kernel routines.

  ROMS_SetInitializeP1   Phase 1 initialization: advertise import and export fields long- and short names.

  ROMS_SetInitializeP2   Phase 2 initialization: Initializes solution, sets application grid arrays, and registers fields into import state and export state.

  ROMS_DataInit          Exports fields during initialization or restart.

  ROMS_SetClock          Sets calendar, start and stop times, and coupling interval.

  ROMS_SetRunClock       Sets ROMS run clock manually.

  ROMS_CheckImport       Checks if the import field is at the correct time.


  ROMS_SetGridArrays     Sets staggered, horizontal grid arrays, grid area, and land/sea masks.

  ROMS_SetStates         Realize fields into import and export states by allocating and initializing pointers.


  ROMS_ModelAdvance      Advances kernel for a coupling interval and calls import and export routines.


  ROMS_SetFinalize       Finalizes execution.

  ROMS_Import            Imports fields from other connected components.

  ROMS_Export            Exports fields to other connected components

MODULE esmf_roms_mod

The ESMF RunSequence configuration file sets how the ESM components are connected and coupled. All the components interact with the same or different coupling time step. Usually, the connector from ROMS to ATM is explicit, whereas the connector from ATM to ROMS is semi-implicit. Often, the timestep of the atmosphere kernel is smaller than that for the ocean. Therefore, it is advantageous for the ATM model to export time-averaged fields over the coupling interval, which is the same as the ROMS timestep. It is semi-implicit because ROMS right-hand-side terms are forced with n+1/2 ATM fields because of the time-averaging. The following diagram shows the explicit and semi-implicit coupling for a DATA-ATM-ROMS system:

For example, in the Hurricane Irene application, we have:

runSeq::
  @*                         # timeStep = wildcard (*), single time loop
    DATA -> WRF              # DATA to WRF connector, step (1)
    DATA
    ROMS -> WRF              # ROMS to WRF connector, step (2)
    WRF                      # step (3)
    WRF -> ROMS              # WRF to ROMS connector, step (4)
    ROMS                     # step (5)
  @
::

---

The test repository is updated to include the Hurricane Irene Test Case. We have the following directory structure for this test:

test/IRENE
          /Data
               /FRC
               /HyCOM
               /NRM
               /OBS
               /ROMS
               /STD
               /WPS
               /WRF
          /Coupling
          /Coupled_RBL4DVAR
          /Forward

Check ​WikiROMS for detailed information about Hurricane Irene Test Case configurations and different execution flavors in the Forward, Coupling, and Coupled_RBL4DVAR sub-directories.

WARNING: This is quite a complex test using several advanced features of ROMS. It is intended for expert users of ROMS. If you a ROMS beginner, I highly recommend staying away from this test. It would be best if you had lots of expertise to run and understand this system. There are Readme files with instructions to run the several flavors for this test case.

---

Many thanks to my collaborators John Wilkin, Julia Levin, Travis Miles, David Robertson, and Joseph Brodie at Rutgers for helping me with several aspects of the configuration and testing of this complex test case. Special thanks to David Robertson for his herculean task of fixing and modernizing COAMPS to compile with aggressive compiler flags for debugging. Also, for his software management skills to compile, link and maintain all these libraries. It requires patience, expertise, and experience! Many thanks to our colleagues at NRL-Monterrey for their help with COAMPS. Finally, many thanks to my colleague Ufuk Turuncoglu for his help with the ESMF/NUOPC coding strategies. This system is inspired by his RegESM system.

This work was funded by the Office of Naval Research (ONR) and the National Oceanic and Atmospheric Administration (NOAA).

[arango]

Loaded ESM/Readme with up-to-date information.