Important: Upated CMEPS NUOPC cap module

[arango]

I implemented the changes made by Ufuk Turunçoğlu to cmeps_roms.F, now called cmeps_roms.h, to avoid interference with the native framework. The native ROMS NUOPC cap module is now called esmf_roms.h. Either NUOPC module is included in esmf_roms.F via CPP. The coupling interface for CDEPS/CMEPS is activated with the CMEPS option, and is intended for connecting to the ​UFS-coastal framework.

For now, the UFS-coastal repository is downloaded from:

git -c submodule.ADCIRC.update=none clone -b feature/coastal_app --recursive https://github.com/oceanmodeling/ufs-coastal

Notice that we are omitting the ADCIRC component during the cloning since it is still private and working with the research branch feature/coastal_app. This coupling capability is still under development and is complex. Thus, it is not ready for the regular user. We are currently testing the interface to connect ROMS to the UFS coupling framework.

An ESM/build_ufs.sh is provided to facilitate configuring and compiling any ROMS application. It is currently configured for the Hurricane Irene (IRENE) application. See ​roms_test for more information. It works for the ROMS-DATA coupling system using the CDEFS connectors or the CMEPS mediator. The DATA component is replacing the atmosphere model, and it tests ROMS connection to CDEPS, CMEPS, and UFS coupling driver.

---

LAPACK/ARPACK Legacy Library:

The ARPACK and its MPI-based parallel (PARPACK) version are well-tested libraries to solve large eigenvalue matrix problems. Our primary use in ROMS is to use its Implicit Restarted Arnoldi Method (IRAM) to solve for the Ritz eigenfunctions in several of our Tangent and Adjoint-based propagators that are part of the Generalized Linear Stability (GLS) package described in Moore et al. (2004) Ocean Modelling paper ​https://doi.org/10.1016/j.ocemod.2003.11.001. As far as I know, we cannot use a modern replacement library. Therefore, we will continue using that library when running any of the drivers of the GLS package.

However, the ARPACK library includes selected Linear Algebra functions from the BLAS and LAPACK libraries. In particular, we use LAPACK's DSTEQR function to solve for the eigenvalues/eigenvectors of the tri-diagonal system used in the various 4D-Var minimizations available in ROMS.

Thus, I spent some time looking for a replacement. It turns out that any modern software out there is built on top of LAPACK (Phyton, MATLAB, R, SciPy, and others). There are also bindings to C and C++. We could use the eigen, written in C++, but it will be complicated since I will have to write and test the inter-operability mechanism in ROMS for C++ to Fortran 2003 and vice versa (Fortran 2003 to C++). It turns out that the libeigen has BLAS and LAPACK components. See: ​https://gitlab.com/libeigen/eigen. On top of that, there are reports that the eigen library is much slower than Fortran libraries. Thus, I am giving up on the eigen library. It is beyond our scope.

I decided to adapt the functions we need from LAPACK into module lapack_mod.F, which I converted to Fortran 2003. It includes a total of 19 LAPACK functions. I modernized the code, which had initially 41 GOTOs. I managed to remove 18 of them. The remaining 23 are located in the function DSTEQR. I was able to get identical 4D-Var solutions during testing. That eigenfunction solver is tricky, and removing the remaining GOTOs is dangerous.

The complicated GOTOs are still in selected functions but can be removed from the final C-preprocessing file (lapack_mod.f90) by activating the CPP option REMOVE_LAPACK_GOTOS. For example, in the function DLASRT used to sort the eigenvalues, I have:

#ifdef REMOVE_LAPACK_GOTOS
            DecLOOP_3 : DO                           ! forever LOOP 3
              DecLOOP_1 : DO                         ! forever LOOP 1
                J = J - 1
                IF (D(J) .GE. DMNMX) EXIT DecLOOP_1  ! terminate LOOP 1
              END DO DecLOOP_1
!
              DecLOOP_2 : DO                         ! forever LOOP 2
                I = I + 1
                IF (D(I) .LE. DMNMX) EXIT DecLOOP_2  ! terminate LOOP 2
              END DO DecLOOP_2
!
              IF (I .LT. J) THEN
                TMP = D(I)
                D(I) = D(J)
                D(J) = TMP
                CYCLE DecLOOP_3                      ! iterate LOOP 3
              ELSE
                EXIT DecLOOP_3                       ! terminate LOOP 3
              END IF
            END DO DecLOOP_3
#else
   60       CONTINUE
   70       CONTINUE
            J = J - 1
            IF (D(J) .LT. DMNMX) GO TO 70
   80       CONTINUE
            I = I + 1
            IF (D(I) .GT. DMNMX) GO TO 80
            IF (I .LT. J) THEN
              TMP = D(I)
              D(I) = D(J)
              D(J) = TMP
              GO TO 60
            END IF
#endif

Notice that the above piece of code is slightly complicated. It requires knowledge of the modern Fortran Standard and Objective-Oriented Programming principles to remove the GOTOs, which were unavailable in Fortran and compilers over 40 years ago. This doesn't imply that the LAPACK library is incorrect and produces the wrong results. I am keeping the original code with the GOTOs for reference and testing.

We no longer need the LAPACK/ARPACK libraries to run any 4D-Var drivers in ROMS because the functions that we need are available in our modern adaptation module lapack_mod.F.

---

Many thanks to Ufuk Turunçoğlu for his hard work and help connecting ROMS to the UFS-coastal.