Description |
This update is very important because it includes changes that allow the user to select the most appropriate MPI library routines for ROMS distribute.F internal message-passing routines: mp_assemble, mp_boundary, mp_collect, and mp_reduce. We can either use high-level (mpi_allgather or mpi_allreduce) or low-level (mpi_isend and mpi_irecv) library routines. In some computers, the low-level routines are more efficient than the high-level routines or vice-versa. There are proprietary MPI libraries offered by the supercomputer and cluster vendors that are optimized and tuned to the computer architecture resulting in very efficient high-order communication routines. In the past, we decided for you and selected the communication methodology. In this update, we no longer make such selections and provide the user with several new C-preprocessing options to choose the appropriate methodology for your computer architecture. The user needs to benchmark their computer(s) and select the most efficient strategy.
Notice that when mpi_allreduce is used, a summation reduction is implied. The array data is stored in a global array (same size) in each parallel tile. The global array is initialized to zero always, and then each MPI node only operates on a different portion of it. Therefore, the computations are carried out by each node; a summation reduction is equivalent to each tile gathering data from other members of the MPI communicator.
Affected communication routines:
- mp_boundary is used to exchange open lateral boundary conditions data between parallel tiles. Each parallel tile is associated with a different MPI node. Two different methods of exchange are currently available using only high-level MPI routines. The user may use C-preprocessing option BOUNDARY_ALLREDUCE to activate exchanges with mpi_allreduce. Otherwise, ROMS will use mpi_allghater as default.
- mp_reduce is used for global reductions according to the operation handle MIN, MAX, or SUM. Various reduction operators are possible when the data is aggregated into a single vector for efficiency. Three different methods are availabe for gathering this type of data using low-level (default) or high-level routines. The user may use C-preprocessing REDUCE_ALLGATHER or REDUCE_ALLREDUCE to use high-level routines mpi_allgather or mpi_allreduce, respectively. If neither option is activated, ROMS will use the low-level functions mpi_isend and mpi_irecv routines to perform the task.
If you want to use the same methodology as in previous versions of the code, check your version of the routine distribute.F and you will see something similar to:
#include "cppdefs.h"
MODULE distribute_mod
#ifdef DISTRIBUTE
# undef ASSEMBLE_ALLGATHER /* use mpi_allgather im mp_assemble */
# undef ASSEMBLE_ALLREDUCE /* use mpi_allreduce in mp_assemble */
# define BOUNDARY_ALLREDUCE /* use mpi_allreduce in mp_boundary */
# undef COLLECT_ALLGATHER /* use mpi_allgather in mp_collect */
# undef COLLECT_ALLREDUCE /* use mpi_allreduce in mp_collect */
# define REDUCE_ALLGATHER /* use mpi_allgather in mp_reduce */
# undef REDUCE_ALLREDUCE /* use mpi_allreduce in mp_reduce */
...
Activate the same C-preprocessing options in your header (*.h) or build script file if you want to use the same set-up.
Usually, in my applications I activate:
#define COLLECT_ALLREDUCE
#define REDUCE_ALLGATHER
which reports to standard output the following information:
!ASSEMBLE_ALL... Using mpi_isend/mpi_recv in mp_assemble routine.
...
!BOUNDARY_ALLGATHER Using mpi_allreduce in mp_boundary routine.
...
COLLECT_ALLREDUCE Using mpi_allreduce in mp_collect routine.
...
REDUCE_ALLGATHER Using mpi_allgather in mp_reduce routine.
I also I made few changes to several of the NetCDF managing routines to avoid opening too many files. The routines netcdf_check_dim and netcdf_inq_var have an additional optional argument ncid to pass the NetCDF file ID, so it is not necessary to open a NetCDF file it is already open.
The second part of this update includes changes to the parallel I/O in ROMS using the NetCDF4 library. Parallel I/O only makes sense in High-Performance Computers (HPC) with the appropriate parallel file system. Otherwise, the I/O data is still processed serially with no improvement on the computation speed up.
In the NetCDF-4 library, the parallel I/O is either done in independent mode or collective mode. In independent mode, each processor accesses the data directly from the file and does not depend on or be affected by other processors. Contrarily in collective mode, all processors participate in doing IO using MPI-I/O and HDF5 for accessing of the tiled data.
To have parallel I/O in ROMS you need to:
|
Description |
There is a format number error in get_wetdry.F:
IF (INI(ng)%ncid.eq.-1) THEN
CALL netcdf_open (ng, model, ncname, 1, INI(ng)%ncid)
IF (FoundError(exit_flag, NoError, __LINE__, &
& __FILE__)) THEN
WRITE (stdout,50) TRIM(ncname)
RETURN
END IF
END IF
Format label 50 is missing.
get_wetdry.f90(62): error #6052: This label has not been defined as a FORMAT label. [50]
WRITE (stdout,50) TRIM(ncname)
------------------------^
|
Description |
This important update corrects few things and enhances information written to standard output:
- Added new module ROMS/Utility/stats.F to compute state field statistics inside a parallel region using routines stats_2dfld and stats_3dfld. Both routines are used to print information from analytical routines (ana_biology, ana_drag, ana_grid, ana_initial, ana_mask, ana_wtype). For example, in the upwelling test case the standard output file now has:
INITIAL: Configuring and initializing forward nonlinear model ...
*******
ANA_GRID - x-location of PSI-points: x_psi
(Grid = 01, Min = 0.00000000E+00 Max = 4.00000000E+04)
ANA_GRID - y-location of PSI-points: y_psi
(Grid = 01, Min = 0.00000000E+00 Max = 8.00000000E+04)
ANA_GRID - x-location of RHO-points: x_rho
(Grid = 01, Min = 5.00000000E+02 Max = 4.05000000E+04)
ANA_GRID - y-location of RHO-points: y_rho
(Grid = 01, Min = -5.00000000E+02 Max = 8.05000000E+04)
ANA_GRID - x-location of U-points: x_u
(Grid = 01, Min = 0.00000000E+00 Max = 4.00000000E+04)
ANA_GRID - y-location of U-points: y_u
(Grid = 01, Min = -5.00000000E+02 Max = 8.05000000E+04)
ANA_GRID - x-location of V-points: x_v
(Grid = 01, Min = 5.00000000E+02 Max = 4.05000000E+04)
ANA_GRID - y-location of V-points: y_v
(Grid = 01, Min = 0.00000000E+00 Max = 8.00000000E+04)
ANA_GRID - reciprocal XI-grid spacing: pm
(Grid = 01, Min = 1.00000000E-03 Max = 1.00000000E-03)
ANA_GRID - reciprocal ETA-grid spacing: pn
(Grid = 01, Min = 1.00000000E-03 Max = 1.00000000E-03)
ANA_GRID - angle between XI-axis and EAST: angler
(Grid = 01, Min = 0.00000000E+00 Max = 0.00000000E+00)
ANA_GRID - Coriolis parameter at RHO-points: f
(Grid = 01, Min = -8.26000000E-05 Max = -8.26000000E-05)
ANA_GRID - bathymetry at RHO-points: h
(Grid = 01, Min = 2.52004887E+01 Max = 1.50000000E+02)
...
ANA_INITIAL - vertically integrated u-momentum component: ubar
(Grid = 01, Min = 0.00000000E+00 Max = 0.00000000E+00)
ANA_INITIAL - vertically integrated v-momentum component: vbar
(Grid = 01, Min = 0.00000000E+00 Max = 0.00000000E+00)
ANA_INITIAL - free-surface: zeta
(Grid = 01, Min = 0.00000000E+00 Max = 0.00000000E+00)
ANA_INITIAL - u-momentum component: u
(Grid = 01, Min = 0.00000000E+00 Max = 0.00000000E+00)
ANA_INITIAL - v-momentum component: v
(Grid = 01, Min = 0.00000000E+00 Max = 0.00000000E+00)
ANA_INITIAL - potential temperature: temp
(Grid = 01, Min = 1.45204834E+01 Max = 2.19365219E+01)
ANA_INITIAL - salinity: salt
(Grid = 01, Min = 3.50000000E+01 Max = 3.50000000E+01)
ANA_BIOLOGY - nitrate concentration: NO3
(Grid = 01, Min = 1.67000000E+00 Max = 2.26358685E+00)
ANA_BIOLOGY - ammonium concentration: NH4
(Grid = 01, Min = 1.00000000E-01 Max = 1.00000000E-01)
ANA_BIOLOGY - chlorophyll concentration: chlorophyll
(Grid = 01, Min = 2.00000000E-02 Max = 2.00000000E-02)
ANA_BIOLOGY - phytoplankton concentration: phytoplankton
(Grid = 01, Min = 8.00000000E-02 Max = 8.00000000E-02)
ANA_BIOLOGY - zooplankton concentration: zooplankton
(Grid = 01, Min = 6.00000000E-02 Max = 6.00000000E-02)
ANA_BIOLOGY - large fraction nitrogen detritus concentration: LdetritusN
(Grid = 01, Min = 2.00000000E-02 Max = 2.00000000E-02)
ANA_BIOLOGY - small fraction nitrogen detritus concentration: SdetritusN
(Grid = 01, Min = 4.00000000E-02 Max = 4.00000000E-02)
ANA_BIOLOGY - large fraction carbon detritus concentration: LdetritusC
(Grid = 01, Min = 2.00000000E-03 Max = 2.00000000E-03)
ANA_BIOLOGY - small fraction carbon detritus concentration: SdetritusC
(Grid = 01, Min = 6.00000000E-02 Max = 6.00000000E-02)
ANA_BIOLOGY - total inorganic carbon: TIC
(Grid = 01, Min = 2.10000000E+03 Max = 2.10000000E+03)
ANA_BIOLOGY - total alkalinity: alkalinity
(Grid = 01, Min = 2.35000000E+03 Max = 2.35000000E+03)
ANA_BIOLOGY - dissolved oxygen concentration: oxygen
(Grid = 01, Min = 4.46229362E+02 Max = 4.46229362E+02)
This information is useful to have. Many thanks to Kate Hedstrom for suggesting this information and Sasha Shchepetkin for his suggestion about the shared-memory interference (race conditions in OpenMP) issues that I was having in stats_2dfld and stats_3dfld.
- Similarly, statistical information is written to standard output from routines get_grid, get_nudgcoef, and get_wetdry.
INITIAL: Configuring and initializing forward nonlinear model ...
*******
GET_GRID - angle between XI-axis and EAST: angle
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 5.38487353E-01 Max = 7.47524111E-01)
GET_GRID - ETA-derivative of inverse metric factor pm: dmde
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = -1.33890089E+00 Max = 2.92981308E-01)
GET_GRID - XI-derivative of inverse metric factor pn: dndx
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = -8.94234719E-01 Max = 2.44594681E-01)
GET_GRID - Coriolis parameter at RHO-points: f
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 7.77782704E-05 Max = 1.05954391E-04)
GET_GRID - bathymetry at RHO-points: h
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 5.00000000E+00 Max = 5.01558186E+03)
GET_GRID - latitude of PSI-points lat_psi
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 3.22838550E+01 Max = 4.65684505E+01)
GET_GRID - latitude of RHO-points lat_rho
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 3.22838550E+01 Max = 4.65684505E+01)
GET_GRID - latitude of U-points: lat_u
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 3.22596780E+01 Max = 4.65941314E+01)
GET_GRID - latitude of V-points: lat_v
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 3.22636115E+01 Max = 4.65856422E+01)
GET_GRID - longitude of PSI-points: lon_psi
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = -8.04626893E+01 Max = -5.97474687E+01)
GET_GRID - longitude of RHO-points: lon_rho
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = -8.04626893E+01 Max = -5.97474687E+01)
GET_GRID - longitude of U-points: lon_u
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = -8.04896326E+01 Max = -5.97260683E+01)
GET_GRID - longitude of V-points: lon_v
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = -8.04917457E+01 Max = -5.97116752E+01)
GET_GRID - mask on PSI-points: mask_psi
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 0.00000000E+00 Max = 2.00000000E+00)
GET_GRID - mask on RHO-points: mask_rho
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 0.00000000E+00 Max = 1.00000000E+00)
GET_GRID - mask on U-points: mask_u
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 0.00000000E+00 Max = 1.00000000E+00)
GET_GRID - mask on V-points: mask_v
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 0.00000000E+00 Max = 1.00000000E+00)
GET_GRID - reciprocal XI-grid spacing: pm
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 1.42195423E-04 Max = 1.45208703E-04)
GET_GRID - reciprocal ETA-grid spacing: pn
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 1.42589316E-04 Max = 1.45608589E-04)
GET_GRID - x-location of PSI-points: x_psi
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 5.10981935E+07 Max = 5.34016198E+07)
GET_GRID - x-location of RHO-points: x-rho
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 5.10919665E+07 Max = 5.34079786E+07)
GET_GRID - x-location of U-points: x_u
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 5.10951975E+07 Max = 5.34039995E+07)
GET_GRID - x-location of V-points: x_v
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 5.10949626E+07 Max = 5.34055999E+07)
GET_GRID - y-location of PSI-points: y-psi
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 6.33431802E+07 Max = 6.54121491E+07)
GET_GRID - y-location of RHO-points: y_rho
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 6.33373400E+07 Max = 6.54190868E+07)
GET_GRID - y-location of U-points: y_u
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 6.33400007E+07 Max = 6.54163037E+07)
GET_GRID - y-location of V-points: y_v
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 6.33405179E+07 Max = 6.54149301E+07)
GET_GRID - quadratic bottom drag coefficient: rdrag2
(Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
(Min = 3.00000000E-03 Max = 3.00000000E-03)
...
GET_NUDGCOEF - potential temperature, inverse nudging coefficients: temp_NudgeCoef
(Grid = 01, File: ../Data/doppio_nudgcoef_7km_deepGS_3dayperim.nc)
(Min = 0.00000000E+00 Max = 3.85802469E-06)
GET_NUDGCOEF - salinity, inverse nudging coefficients: salt_NudgeCoef
(Grid = 01, File: ../Data/doppio_nudgcoef_7km_deepGS_3dayperim.nc)
(Min = 0.00000000E+00 Max = 3.85802469E-06)
- Modified several routines to provide more information when an input file is missing from ocean.in. In the past, we use to get the following error:
Found Error: ** Line: 6565 Source: ROMS/Modules/mod_netcdf.F, netcdf_open
NETCDF_OPEN - unable to open existing NetCDF file:
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ... @^@^@
call from: ROMS/Utility/inp_par.F, find_file
Now, we get better information about the error:
INP_PAR:FIND_FILE - empty file name string for standard input script
KeyWord: TIDENAME
Found Error: 05 Line: 1396 Source: ROMS/Utility/inp_par.F
READ_PHYPAR - Grid 01, could not find input file: Oops unassigned file name.
Check standard input script...
Found Error: 04 Line: 5674 Source: ROMS/Utility/read_phypar.F
Found Error: 04 Line: 139 Source: ROMS/Utility/inp_par.F
Found Error: 04 Line: 111 Source: ROMS/Drivers/nl_ocean.h
This information is more useful and tell me that the file name for keyWord TIDENAME is missing from ocean.in and this input script needs to be updated to the lastest version.
- Modified several routines, so the diagnostic printed to standard output have identical numbers in serial, serial with partitions, parallel distributed-memory (MPI), and parallel shared-memory (OpenMP) applications. Several of the global parallel reduction CRITICAL sections were updated.
|