NUOPC Cap

ROMS Native ESMF/NUOPC Cap

Overview

This document describes the ROMS ESMF/NUOPC cap module Master/esmf_roms.F used in the ROMS native coupling infrastructure shown below. In this case, ROMS is driving the coupled system (Master/esmf_esm.F), which sets the NOUPC-based generic services for each Earth System Model (ESM) component (data, atmosphere, sea-ice, wave, and ocean), the coupler connectors, and the RunSequence policy. It also provides the NUOPC cap modules for each coupled component. Detailed information about ESMF/NUOPC and how it is implemented in ROMS can be found on the Earth System Modeling Framework WikiROMS page.

A NUOPC cap is a Fortran module that serves as the interface to a model when used in a NUOPC-based coupled system. The term cap is used because it is a lightweight software layer that sits on top of the model kernel, making calls to its initialize, run, and finalize phases and exposing the model data structures to the field exchange (import and export states) between coupled components.

Figure 1: ROMS native ESMF/NUOPC coupling infrastructure

Implementation

The Master/mod_esmf_esm.F module declares several derived-type structures to facilitate the management of all internal objects and variables in each ESM coupled component:

• ESM coupling time managing variables and ESMF objects.
  TYPE :: ESM_Clock
  
  logical :: Restarted
  
  integer (i8b) :: AdvanceCount ! advance counter
  
  real (dp) :: Current_Time ! seconds
  real (dp) :: Time_Reference ! seconds
  real (dp) :: Time_Restart ! seconds
  real (dp) :: Time_Start ! seconds
  real (dp) :: Time_Stop ! seconds
  real (dp) :: Time_Step ! seconds
  
  character (len=22) :: Name
  character (len=22) :: CalendarString ! 360_day, gregorian
  character (len=22) :: Time_ReferenceString
  character (len=22) :: Time_RestartString
  character (len=22) :: Time_StartString
  character (len=22) :: Time_StopString
  
  TYPE (ESMF_Calendar)  :: Calendar
  TYPE (ESMF_Clock)  :: Clock
  TYPE (ESMF_Direction_flag) :: Direction
  TYPE (ESMF_Time)  :: CurrentTime
  TYPE (ESMF_Time)  :: ReferenceTime
  TYPE (ESMF_Time)  :: RestartTime
  TYPE (ESMF_Time)  :: StartTime
  TYPE (ESMF_Time)  :: StopTime
  TYPE (ESMF_TimeInterval)  :: TimeStep
  
  END TYPE ESM_Clock
  
  . . .
  
  TYPE (ESM_Clock), allocatable, target :: ClockInfo(:)

• ESM coupled state sets. If appropriate, it includes the logic for connecting nested grids.
  TYPE :: ESM_CplSet
  
  logical, allocatable :: LinkedGrid(:,:) ! connected grid
  
  logical, allocatable :: DataCoupledSets(:,:) ! DATA linked sets
  
  character (len=100), allocatable :: SetLabel(:) ! set label
  character (len=100), allocatable :: ExpLabel(:) ! export label
  character (len=100), allocatable :: ImpLabel(:) ! import label
  
  TYPE (ESMF_State), allocatable :: ExportState(:,:) ! export set
  
  END TYPE ESM_CplSet
  
  TYPE (ESM_CplSet), allocatable, target :: COUPLED(:)

• DATA model field processing information.
  TYPE ESM_Data
  
  logical :: Lcycle ! cycling time coordinate
  logical :: Lcoord ! coordinates attribute
  logical :: Lmask ! land/sea mask
  logical :: Lmulti ! field across multi-files
  logical :: LastRec ! processed last record
  
  integer :: Icomp ! target component index
  integer :: ncid ! NetCDF file ID
  integer :: Tid ! time variable ID
  integer :: Vid ! field variable ID
  integer :: Nvdim ! number spatial dimensions
  integer :: Zlevel ! level index to process
  integer :: Nrec ! number of time records
  integer :: Trec ! latest read time record
  integer :: Tindex ! rolling two-time indices
  integer :: LandValue ! Masking land value
  integer :: SeaValue ! Masking sea value
  
  integer, allocatable :: Vsize(:) ! dimensions size
  
  real(dp) :: add_offset ! add_offset attribute
  real(dp) :: FillValue ! _FillValue attribute
  real(dp) :: scale_factor ! scale_factor attribute
  real(dp) :: Clength ! time cycling length
  real(r8) :: LonMin ! grid minimum longitude
  real(r8) :: LonMax ! grid maximum longitude
  real(r8) :: LatMin ! grid minimum latitude
  real(r8) :: LatMax ! grid maximum latitude
  real(dp) :: Tscale ! time scale to day
  real(dp) :: Tmono ! monotonic time (days)
  real(dp) :: Tmin ! time minimum value
  real(dp) :: Tmax ! time maximum value
  real(dp) :: Tstr ! lower time-snapshot
  real(dp) :: Tend ! upper time-snapshot
  real(r8) :: Vmin ! variable minimum value
  real(r8) :: Vmax ! variable maximum value
  real(dp) :: Tintrp(2) ! interpolation time (days)
  real(dp) :: Vtime(2) ! latest two-time values
  real(dp) :: Date(6,2) ! time-snapshots dates YYYY,MM,DD hh:mm:ss.ss
  
  character(len=30), allocatable :: Dname(:) ! variable dimensions names
  character(len=20), allocatable :: Vcoord(:) ! variable coordinates names
  
  character(len=22 ) :: DateString(2) ! date-snapshots string
  character(len=30 ) :: SpecialAction ! special processing
  character(len=100) :: Tname ! time variable name
  character(len=100) :: Tunits ! time variable units
  character(len=100) :: Vname ! variable name
  character(len=100) :: Vunits ! variable units
  character(len=256) :: Vdescriptor ! Variable descriptive name
  character(len=256) :: Vlongname ! long_name attribute
  character(len=256) :: ncfile ! NetCDF filename
  
  real(r8), allocatable :: lon(:,:) ! field longitude
  real(r8), allocatable :: lat(:,:) ! field latitude
  real(r8), allocatable :: mask(:,:) ! field land/sea mask
  
  real(r8), allocatable :: A2d(:,:) ! time interpolated 2D data
  real(r8), allocatable :: A3d(:,:,:) ! time interpolated 3D data
  real(r8), allocatable :: A2dG(:,:,:) ! latest 2D data snapsnots
  real(r8), allocatable :: A3dG(:,:,:,:)! latest 3D data snapsnots
  
  TYPE (ESMF_Field)  :: field ! field object
  TYPE (ESMF_Grid)  :: grid ! field grid object
  TYPE (ESMF_RouteHandle) :: rhandle ! field RouteHandle
  
  END TYPE ESM_Data

• Import and export fields metadata information.
  TYPE :: ESM_Field
  
  logical :: connected ! connected to coupler
  logical :: debug_write ! write exchanged field
  logical :: enable_integral_adj ! area integral adjusted
  
  integer :: fid ! internal field ID
  integer :: gtype ! field grid mesh type
  integer :: etype ! field extrapolation flag
  integer :: itype ! field interpolation flag
  integer :: Tindex ! rolling two-time indices
  
  character (len=20) :: Ctarget ! component destination
  character (len=22) :: DateString(2) ! date-snapshots string
  
  character (len=:), allocatable :: short_name ! short name
  character (len=:), allocatable :: standard_name ! standard name
  character (len=:), allocatable :: long_name ! long name
  character (len=:), allocatable :: dst_gtype ! DST grid type
  character (len=:), allocatable :: dst_units ! DST units
  character (len=:), allocatable :: src_gtype ! SRC grid type
  character (len=:), allocatable :: src_units ! SRC units
  character (len=:), allocatable :: nc_vname ! DATA Vname
  character (len=:), allocatable :: nc_tname ! DATA Tname
  character (len=:), allocatable :: RegridMethod ! regrid method
  character (len=:), allocatable :: ExtrapMethod ! extrapolate
  
  real (dp) :: scale_factor ! field scale factor
  real (dp) :: add_offset ! field add offset value
  real (dp) :: Tmin ! DATA time minimum value
  real (dp) :: Tmax ! DATA time maximum value
  real (dp) :: Tstr ! DATA lower time-snapshot
  real (dp) :: Tend ! DATA upper time-snapshot
  real (dp) :: Tintrp(2) ! interpolation time (day)
  real (dp) :: Vtime(2) ! latest two-time values
  
  TYPE (ESMF_RouteHandle) :: rhandle  ! field RouteHandle
  
  END TYPE ESM_Field

• Import and export fields mesh data.
  TYPE :: ESM_Mesh
  
  integer :: gid ! grid ID
  integer :: gtype ! grid mesh type
  
  integer (i4b), allocatable :: mask(:,:) ! grid land/sea mask
  
  real (r8), allocatable :: lon(:,:) ! grid longitude
  real (r8), allocatable :: lat(:,:) ! grid latitude
  real (r8), allocatable :: area(:,:) ! grid area
  
  END TYPE ESM_Mesh

• Coupled models high-level object, [Nmodels=1].
  TYPE :: ESM_Model
  
  logical :: IsActive ! active for coupling
  
  integer (i4b) :: LandValue ! land mask value
  integer (i4b) :: SeaValue ! sea mask value
  
  integer :: Ngrids ! number nested grids
  
  integer :: ExportCalls ! export CALL counter
  integer :: ImportCalls ! import CALL counter
  
  integer :: nPETs ! number model PETs
  integer, allocatable :: PETlist(:) ! component PETs list
  
  integer, allocatable :: TimeFrac(:,:) ! driver time fraction
  
  character (len=:), allocatable :: name ! component name
  
  TYPE (ESMF_Grid), allocatable :: grid(:) ! grid object
  TYPE (ESM_Mesh), allocatable :: mesh(:) ! mesh
  TYPE (ESM_Field), allocatable :: ImportField(:) ! import fields
  TYPE (ESM_Field), allocatable :: ExportField(:) ! export fields
  TYPE (ESMF_State), allocatable :: ImportState(:) ! import state
  TYPE (ESMF_State), allocatable :: ExportState(:) ! export state
  
  END TYPE ESM_Model
  
  TYPE (ESM_Model), allocatable, target :: MODELS(:)

• Internal module parameters and variables:
  
  • Switch to trace/track run sequence during debugging. All information is written to Fortran unit trac. For now, it uses the standard output unit.
    logical :: ESM_track = .TRUE. ! trace/track CALL sequence switch
  • Number of coupled ESM gridded components. Currently, five ESM components are supported (ROMS, DATA, ATMOSPHERE, SEA-ICE, and WAVE model types).
    integer, parameter :: Nmodels = 5
  • Number of nested grids. All are initialized to just one grid; its values are overwritten during processing. An additional variable NgridsR is created to avoid using ROMS module mod_param in the generic interface. Both Ngrids and NgridsR will have the same value.
    integer, parameter :: NgridsA = 1 ! Atmosphere Model
    integer, parameter :: NgridsD = 1 ! DATA Model
    integer, parameter :: NgridsI = 1 ! Sea-ice Model
    integer, parameter :: NgridsR = 1 ! ROMS
    integer, parameter :: NgridsW = 1 ! Wave Model
  • Coupled models identification indices.
    integer, parameter :: Idriver = 0
    integer, parameter :: Iroms = 1
    integer, parameter :: Iatmos = 2
    integer, parameter :: Idata = 3
    integer, parameter :: Iseaice = 4
    integer, parameter :: Iwave = 5
  • Number of ROMS export and import fields per component.
    integer, allocatable :: Nexport(:)
    integer, allocatable :: Nimport(:)
  • Model coupling type: [1] Explicit, [otherwise] Semi-Implicit. In explicit coupling, exchange fields at the next time-step are defined using known values from the time-step before it. Explicit methods require less computational effort and are accurate for small coupling time-steps. In implicit coupling, exchange fields at the next time-step are defined by including values at the next time-step. Implicit methods are stable and allow longer coupling time-steps but are more computationally expensive. In semi-implicit coupling, ROMS -> ATM is explicit, ATM -> ROMS is implicit.
    integer :: CouplingType = 1
  • Distributed-memory communicator handle for each component, Total number of PETs needed in concurrent PET layout, rank of each PET, and PET layout (sequential or concurrent).
    integer, allocatable :: ESMcomm(:)
    integer :: sumPETs
    integer :: PETrank
    character (len=10) :: PETlayoutOption
  • Coupling debugging flag: [0] no debugging, [1] reports informative messages, or [2] '1' plus writes grid information in VTK format, [3] '2' plus writes exchage fields into NetCDF files.
    integer :: DebugLevel = 0
  • Execution tracing level flag: [0] no tracing, [1] reports sequence of coupling subroutine calls, or [2] <1> plus writes voluminous ESMF library tracing information which slowdown performace, and creates large log file.
    integer :: TraceLevel = 0
  • Standard output units and log messages filename for coupler and ESMF library.
    integer :: cplout = 77 ! coupling driver
    integer :: dataout = 77 ! data component
    
    character (len= 8), parameter :: ESMnameLog = 'log.esmf'
    character (len= 8), parameter :: CouplerLog = 'log.coupler'
  • Coupled model staggered grid-cell type indices.
    integer, parameter :: Inan = 0 ! unstaggered, cell center
    integer, parameter :: Icenter = 1 ! cell center
    integer, parameter :: Icorner = 2 ! cell corners
    integer, parameter :: Iupoint = 3 ! right and left cell faces
    integer, parameter :: Ivpoint = 4 ! upper and lower cell faces
    
    character (len=6), dimension(0:4) :: GridType = &
    & (/ 'N/A ', &
    & 'Center', &
    & 'Corner', &
    & 'U ', &
    & 'V ' /)
  • Standard input filename for each coupled model, [Nmodels].
    character (len=256), allocatable :: INPname(:)
  • ROMS coupling YAML configuration filename.
    character (len=256), allocatable :: CPLname
  • Is there an equivalant to this in the Native??? ESM strings.
    character (len=:), allocatable :: CoupledSet
    character (len=:), allocatable :: ExportStateName
    character (len=:), allocatable :: ImportStateName
  • ESM single and double precision constants.
    integer (i4b), parameter :: MAPPED_MASK = 99_i4b
    integer (i4b), parameter :: UNMAPPED_MASK = 98_i4b
    
    real (dp), parameter :: MISSING_dp = 1.0E20_dp
    real (r4), parameter :: MISSING_r4 = 1.0E20_r4
    real (r8), parameter :: MISSING_r8 = 1.0E20_r8
    
    real (dp), parameter :: TOL_dp = 0.5E20_dp
    real (r4), parameter :: TOL_r4 = 0.5E20_r4
    real (r8), parameter :: TOL_r8 = 0.5E20_r8

Capabilities

The ROMS cap module contains a set of subroutines that are required by NUOPC. These subroutines are called by the NUOPC infrastructure according to a predefined calling sequence. Some subroutines are called during the initialization of the coupled system, some during the run of the coupled system, and some during the finalization of the coupled system.

The initialization sequence is the most complex and is governed by the NUOPC technical rules. Details about the initialization sequence can be found in the NUOPC Reference Manual. The ROMS cap requires ESMF version 8 or higher.

ROMS_SetServices	Entry point to the ROMS cap and the only public routine. It sets the ROMS component chared-object entry points for using NUOPC generic methods for initialize, run, and finalize.
ROMS_SetInitializeP1	ROMS component phase 1 initialization which sets import and export fields long and short names into its respective state.
ROMS_SetInitializeP2	ROMS component phase 2 initialization which initializes the ROMS component (ROMS_initialize), sets component grid (ROMS_SetGridArrays), and adds fields into import and export into respective states.
ROMS_DataInit	Exports ROMS component fields during initialization or restart.
ROMS_SetClock	Sets ROMS component date calendar, start and stop times, and coupling interval. At initialization, the variable tdays is the initial time measured in fractional days since the reference time.
ROMS_SetRunClock	Sets ROMS run clock manually to avoid getting zero time stamps at the first regridding call.
ROMS_CheckImport	Checks if ROMS component import field is at the correct time.
ROMS_SetGridArrays	Sets ROMS component staggered, horizontal grid arrays, grid area, and land/sea mask if any.
ROMS_SetStates	Adds ROMS component export and import fields into its respective state.
ROMS_ModelAdvance	Advances ROMS component for a coupling interval. It calls ROMS_import and ROMS_export routines.
ROMS_SetFinalize	Finalizes ROMS component execution.
ROMS_Import	Imports fields into ROMS. The fields are loaded into the snapshot storage arrays to allow time interpolation elsewhere.
ROMS_Export	Exports ROMS fields to other gridded components.