Custom Query (972 matches)

Filters
 
Or
 
  
 
Columns

Show under each result:


Results (769 - 771 of 972)

Ticket Owner Reporter Resolution Summary
#747 arango Done Important: Updating MPI communication options and NetCDF-4 Parallel I/O
Description

This update is very important because it includes changes that allow the user to select the most appropriate MPI library routines for ROMS distribute.F internal message-passing routines: mp_assemble, mp_boundary, mp_collect, and mp_reduce. We can either use high-level (mpi_allgather or mpi_allreduce) or low-level (mpi_isend and mpi_irecv) library routines. In some computers, the low-level routines are more efficient than the high-level routines or vice-versa. There are proprietary MPI libraries offered by the supercomputer and cluster vendors that are optimized and tuned to the computer architecture resulting in very efficient high-order communication routines. In the past, we decided for you and selected the communication methodology. In this update, we no longer make such selections and provide the user with several new C-preprocessing options to choose the appropriate methodology for your computer architecture. The user needs to benchmark their computer(s) and select the most efficient strategy.

Notice that when mpi_allreduce is used, a summation reduction is implied. The array data is stored in a global array (same size) in each parallel tile. The global array is initialized to zero always, and then each MPI node only operates on a different portion of it. Therefore, the computations are carried out by each node; a summation reduction is equivalent to each tile gathering data from other members of the MPI communicator.

Affected communication routines:

  • mp_assemble is used in the nesting algorithms to gather data from all the MPI communicator members (parallel tiles) for the expensive two-way exchange between nested grids. There are three different methods for gathering data using low-level (default) or high-level routines. The user may use C-preprocessing ASSEMBLE_ALLGATHER or ASSEMBLE_ALLREDUCE to use high-level routines mpi_allgather or mpi_allreduce, respectively. If neither option is activated, ROMS will use the low-level functions mpi_isend and mpi_irecv routines to perform the task.
         INTERFACE mp_assemble
           MODULE PROCEDURE mp_assemblef_1d
           MODULE PROCEDURE mp_assemblef_2d
           MODULE PROCEDURE mp_assemblef_3d
           MODULE PROCEDURE mp_assemblei_1d
           MODULE PROCEDURE mp_assemblei_2d
         END INTERFACE mp_assemble
    
  • mp_boundary is used to exchange open lateral boundary conditions data between parallel tiles. Each parallel tile is associated with a different MPI node. Two different methods of exchange are currently available using only high-level MPI routines. The user may use C-preprocessing option BOUNDARY_ALLREDUCE to activate exchanges with mpi_allreduce. Otherwise, ROMS will use mpi_allghater as default.
  • mp_collect is used for gathering data for 1D integer or floating-point arrays. It is used in several ROMS algorithms: floats, stations, 4D-Var observation, and parallel I/O. Again, there are three different methods for gathering this type of data using low-level (default) or high-level routines. The user may use C-preprocessing COLLECT_ALLGATHER or COLLECT_ALLREDUCE to use high-level routines mpi_allgather or mpi_allreduce, respectively. If neither option is activated, ROMS will use the low-level functions mpi_isend and mpi_irecv routines to perform the task.
         INTERFACE mp_collect
           MODULE PROCEDURE mp_collect_f
           MODULE PROCEDURE mp_collect_i
         END INTERFACE mp_collect
    
  • mp_reduce is used for global reductions according to the operation handle MIN, MAX, or SUM. Various reduction operators are possible when the data is aggregated into a single vector for efficiency. Three different methods are availabe for gathering this type of data using low-level (default) or high-level routines. The user may use C-preprocessing REDUCE_ALLGATHER or REDUCE_ALLREDUCE to use high-level routines mpi_allgather or mpi_allreduce, respectively. If neither option is activated, ROMS will use the low-level functions mpi_isend and mpi_irecv routines to perform the task.

If you want to use the same methodology as in previous versions of the code, check your version of the routine distribute.F and you will see something similar to:

#include "cppdefs.h"
      MODULE distribute_mod
#ifdef DISTRIBUTE

# undef  ASSEMBLE_ALLGATHER /* use mpi_allgather im mp_assemble */
# undef  ASSEMBLE_ALLREDUCE /* use mpi_allreduce in mp_assemble */
# define BOUNDARY_ALLREDUCE /* use mpi_allreduce in mp_boundary */
# undef  COLLECT_ALLGATHER  /* use mpi_allgather in mp_collect  */
# undef  COLLECT_ALLREDUCE  /* use mpi_allreduce in mp_collect  */
# define REDUCE_ALLGATHER   /* use mpi_allgather in mp_reduce   */
# undef  REDUCE_ALLREDUCE   /* use mpi_allreduce in mp_reduce   */
...

Activate the same C-preprocessing options in your header (*.h) or build script file if you want to use the same set-up.

Usually, in my applications I activate:

#define COLLECT_ALLREDUCE
#define REDUCE_ALLGATHER

which reports to standard output the following information:

!ASSEMBLE_ALL...    Using mpi_isend/mpi_recv in mp_assemble routine.
...
!BOUNDARY_ALLGATHER Using mpi_allreduce in mp_boundary routine.
...
COLLECT_ALLREDUCE   Using mpi_allreduce in mp_collect routine.
...
REDUCE_ALLGATHER    Using mpi_allgather in mp_reduce routine.

I also I made few changes to several of the NetCDF managing routines to avoid opening too many files. The routines netcdf_check_dim and netcdf_inq_var have an additional optional argument ncid to pass the NetCDF file ID, so it is not necessary to open a NetCDF file it is already open.


The second part of this update includes changes to the parallel I/O in ROMS using the NetCDF4 library. Parallel I/O only makes sense in High-Performance Computers (HPC) with the appropriate parallel file system. Otherwise, the I/O data is still processed serially with no improvement on the computation speed up.

In the NetCDF-4 library, the parallel I/O is either done in independent mode or collective mode. In independent mode, each processor accesses the data directly from the file and does not depend on or be affected by other processors. Contrarily in collective mode, all processors participate in doing IO using MPI-I/O and HDF5 for accessing of the tiled data.

To have parallel I/O in ROMS you need to:

  • Activate C-preprocessing option HDF5 and PARALLEL_IO.
  • Compile with the parallel NetCDF4 library. If using the build script make sure that USE_PARALLEL_IO is activated, so the appropriate library is selected.
  • Make sure that all input NetCDF files are of type netCDF-4. Use ncdump -k filename.nc to inquire about the file type:
    % ncdump -k grid_doppio.nc
    netCDF-4
    
  • Convert input files to netCDF-4, if necessary. Use the following command to convert from classic or 64-bit offset:
    ncccopy -k 'netCDF-4' filetype3.nc filetype4.nc
    or
    nccopy -4 filetype3.nc  filetype4.nc
    
    Make sure that the nccopy version is that from the NetCDF-4 library.
#748 kate Fixed get_wetdry.F missing label 50
Description

There is a format number error in get_wetdry.F:

      IF (INI(ng)%ncid.eq.-1) THEN
        CALL netcdf_open (ng, model, ncname, 1, INI(ng)%ncid)
        IF (FoundError(exit_flag, NoError, __LINE__,                    &
     &                 __FILE__)) THEN
          WRITE (stdout,50) TRIM(ncname)
          RETURN
        END IF
      END IF

Format label 50 is missing.

get_wetdry.f90(62): error #6052: This label has not been defined as a FORMAT label.   [50]
          WRITE (stdout,50) TRIM(ncname)
------------------------^

#749 arango Done Important: Miscelaneous Update: GLS and standard ouput information
Description

This important update corrects few things and enhances information written to standard output:

  • Corrected a bug in the drivers afte_ocean.h and fte_ocean.h (Finite Time Eigenmodes algorithms). We were getting segmentation violation after the convergence. The work array used to compute the complex eigenvectors was wrong. We need to use SworkR instead of SworkD. For example, we need to have:
                        state(ng)%vector => STORAGE(ng)%Rvector(Is:Ie,i)
                        ad_state(ng)%vector => SworkR(Is:Ie)
    instead of 
                        state(ng)%vector => STORAGE(ng)%Rvector(Is:Ie,i)
                        ad_state(ng)%vector => STORAGE(ng)%SworkD(Is:Ie)
    
    Many thanks to Andy Moore for bringing this to my attention.
  • Added new module ROMS/Utility/stats.F to compute state field statistics inside a parallel region using routines stats_2dfld and stats_3dfld. Both routines are used to print information from analytical routines (ana_biology, ana_drag, ana_grid, ana_initial, ana_mask, ana_wtype). For example, in the upwelling test case the standard output file now has:
    INITIAL: Configuring and initializing forward nonlinear model ...
     *******
    
        ANA_GRID    - x-location of PSI-points: x_psi
                       (Grid = 01, Min =  0.00000000E+00 Max =  4.00000000E+04)
        ANA_GRID    - y-location of PSI-points: y_psi
                       (Grid = 01, Min =  0.00000000E+00 Max =  8.00000000E+04)
        ANA_GRID    - x-location of RHO-points: x_rho
                       (Grid = 01, Min =  5.00000000E+02 Max =  4.05000000E+04)
        ANA_GRID    - y-location of RHO-points: y_rho
                       (Grid = 01, Min = -5.00000000E+02 Max =  8.05000000E+04)
        ANA_GRID    - x-location of U-points: x_u
                       (Grid = 01, Min =  0.00000000E+00 Max =  4.00000000E+04)
        ANA_GRID    - y-location of U-points: y_u
                       (Grid = 01, Min = -5.00000000E+02 Max =  8.05000000E+04)
        ANA_GRID    - x-location of V-points: x_v
                       (Grid = 01, Min =  5.00000000E+02 Max =  4.05000000E+04)
        ANA_GRID    - y-location of V-points: y_v
                       (Grid = 01, Min =  0.00000000E+00 Max =  8.00000000E+04)
        ANA_GRID    - reciprocal XI-grid spacing: pm
                       (Grid = 01, Min =  1.00000000E-03 Max =  1.00000000E-03)
        ANA_GRID    - reciprocal ETA-grid spacing: pn
                       (Grid = 01, Min =  1.00000000E-03 Max =  1.00000000E-03)
        ANA_GRID    - angle between XI-axis and EAST: angler
                       (Grid = 01, Min =  0.00000000E+00 Max =  0.00000000E+00)
        ANA_GRID    - Coriolis parameter at RHO-points: f
                       (Grid = 01, Min = -8.26000000E-05 Max = -8.26000000E-05)
        ANA_GRID    - bathymetry at RHO-points: h
                       (Grid = 01, Min =  2.52004887E+01 Max =  1.50000000E+02)
    
    ...
    
        ANA_INITIAL - vertically integrated u-momentum component: ubar
                       (Grid = 01, Min =  0.00000000E+00 Max =  0.00000000E+00)
        ANA_INITIAL - vertically integrated v-momentum component: vbar
                       (Grid = 01, Min =  0.00000000E+00 Max =  0.00000000E+00)
        ANA_INITIAL - free-surface: zeta
                       (Grid = 01, Min =  0.00000000E+00 Max =  0.00000000E+00)
        ANA_INITIAL - u-momentum component: u
                       (Grid = 01, Min =  0.00000000E+00 Max =  0.00000000E+00)
        ANA_INITIAL - v-momentum component: v
                       (Grid = 01, Min =  0.00000000E+00 Max =  0.00000000E+00)
        ANA_INITIAL - potential temperature: temp
                       (Grid = 01, Min =  1.45204834E+01 Max =  2.19365219E+01)
        ANA_INITIAL - salinity: salt
                       (Grid = 01, Min =  3.50000000E+01 Max =  3.50000000E+01)
        ANA_BIOLOGY - nitrate concentration: NO3
                       (Grid = 01, Min =  1.67000000E+00 Max =  2.26358685E+00)
        ANA_BIOLOGY - ammonium concentration: NH4
                       (Grid = 01, Min =  1.00000000E-01 Max =  1.00000000E-01)
        ANA_BIOLOGY - chlorophyll concentration: chlorophyll
                       (Grid = 01, Min =  2.00000000E-02 Max =  2.00000000E-02)
        ANA_BIOLOGY - phytoplankton concentration: phytoplankton
                       (Grid = 01, Min =  8.00000000E-02 Max =  8.00000000E-02)
        ANA_BIOLOGY - zooplankton concentration: zooplankton
                       (Grid = 01, Min =  6.00000000E-02 Max =  6.00000000E-02)
        ANA_BIOLOGY - large fraction nitrogen detritus concentration: LdetritusN
                       (Grid = 01, Min =  2.00000000E-02 Max =  2.00000000E-02)
        ANA_BIOLOGY - small fraction nitrogen detritus concentration: SdetritusN
                       (Grid = 01, Min =  4.00000000E-02 Max =  4.00000000E-02)
        ANA_BIOLOGY - large fraction carbon detritus concentration: LdetritusC
                       (Grid = 01, Min =  2.00000000E-03 Max =  2.00000000E-03)
        ANA_BIOLOGY - small fraction carbon detritus concentration: SdetritusC
                       (Grid = 01, Min =  6.00000000E-02 Max =  6.00000000E-02)
        ANA_BIOLOGY - total inorganic carbon: TIC
                       (Grid = 01, Min =  2.10000000E+03 Max =  2.10000000E+03)
        ANA_BIOLOGY - total alkalinity: alkalinity
                       (Grid = 01, Min =  2.35000000E+03 Max =  2.35000000E+03)
        ANA_BIOLOGY - dissolved oxygen concentration: oxygen
                       (Grid = 01, Min =  4.46229362E+02 Max =  4.46229362E+02)
    
    This information is useful to have. Many thanks to Kate Hedstrom for suggesting this information and Sasha Shchepetkin for his suggestion about the shared-memory interference (race conditions in OpenMP) issues that I was having in stats_2dfld and stats_3dfld.
  • Similarly, statistical information is written to standard output from routines get_grid, get_nudgcoef, and get_wetdry.
    INITIAL: Configuring and initializing forward nonlinear model ...
     *******
    
        GET_GRID    - angle between XI-axis and EAST: angle
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  5.38487353E-01 Max =  7.47524111E-01)
        GET_GRID    - ETA-derivative of inverse metric factor pm: dmde
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min = -1.33890089E+00 Max =  2.92981308E-01)
        GET_GRID    - XI-derivative of inverse metric factor pn: dndx
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min = -8.94234719E-01 Max =  2.44594681E-01)
        GET_GRID    - Coriolis parameter at RHO-points: f
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  7.77782704E-05 Max =  1.05954391E-04)
        GET_GRID    - bathymetry at RHO-points: h
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  5.00000000E+00 Max =  5.01558186E+03)
        GET_GRID    - latitude of PSI-points lat_psi
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  3.22838550E+01 Max =  4.65684505E+01)
        GET_GRID    - latitude of RHO-points lat_rho
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  3.22838550E+01 Max =  4.65684505E+01)
        GET_GRID    - latitude of U-points: lat_u
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  3.22596780E+01 Max =  4.65941314E+01)
        GET_GRID    - latitude of V-points: lat_v
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  3.22636115E+01 Max =  4.65856422E+01)
        GET_GRID    - longitude of PSI-points: lon_psi
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min = -8.04626893E+01 Max = -5.97474687E+01)
        GET_GRID    - longitude of RHO-points: lon_rho
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min = -8.04626893E+01 Max = -5.97474687E+01)
        GET_GRID    - longitude of U-points: lon_u
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min = -8.04896326E+01 Max = -5.97260683E+01)
        GET_GRID    - longitude of V-points: lon_v
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min = -8.04917457E+01 Max = -5.97116752E+01)
        GET_GRID    - mask on PSI-points: mask_psi
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  0.00000000E+00 Max =  2.00000000E+00)
        GET_GRID    - mask on RHO-points: mask_rho
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  0.00000000E+00 Max =  1.00000000E+00)
        GET_GRID    - mask on U-points: mask_u
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  0.00000000E+00 Max =  1.00000000E+00)
        GET_GRID    - mask on V-points: mask_v
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  0.00000000E+00 Max =  1.00000000E+00)
        GET_GRID    - reciprocal XI-grid spacing: pm
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  1.42195423E-04 Max =  1.45208703E-04)
        GET_GRID    - reciprocal ETA-grid spacing: pn
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  1.42589316E-04 Max =  1.45608589E-04)
        GET_GRID    - x-location of PSI-points: x_psi
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  5.10981935E+07 Max =  5.34016198E+07)
        GET_GRID    - x-location of RHO-points: x-rho
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  5.10919665E+07 Max =  5.34079786E+07)
        GET_GRID    - x-location of U-points: x_u
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  5.10951975E+07 Max =  5.34039995E+07)
        GET_GRID    - x-location of V-points: x_v
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  5.10949626E+07 Max =  5.34055999E+07)
        GET_GRID    - y-location of PSI-points: y-psi
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  6.33431802E+07 Max =  6.54121491E+07)
        GET_GRID    - y-location of RHO-points: y_rho
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  6.33373400E+07 Max =  6.54190868E+07)
        GET_GRID    - y-location of U-points: y_u
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  6.33400007E+07 Max =  6.54163037E+07)
        GET_GRID    - y-location of V-points: y_v
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  6.33405179E+07 Max =  6.54149301E+07)
        GET_GRID    - quadratic bottom drag coefficient: rdrag2
                       (Grid = 01, File: ../Data/grid_doppio_7km_sponge_v3.nc)
                       (Min =  3.00000000E-03 Max =  3.00000000E-03)
    
    ...
    
       GET_NUDGCOEF - potential temperature, inverse nudging coefficients: temp_NudgeCoef
                       (Grid = 01, File: ../Data/doppio_nudgcoef_7km_deepGS_3dayperim.nc)
                       (Min =  0.00000000E+00 Max =  3.85802469E-06)
       GET_NUDGCOEF - salinity, inverse nudging coefficients: salt_NudgeCoef
                       (Grid = 01, File: ../Data/doppio_nudgcoef_7km_deepGS_3dayperim.nc)
                       (Min =  0.00000000E+00 Max =  3.85802469E-06)
    
  • Modified several routines to provide more information when an input file is missing from ocean.in. In the past, we use to get the following error:
    Found Error: **   Line: 6565     Source: ROMS/Modules/mod_netcdf.F, netcdf_open
    
     NETCDF_OPEN - unable to open existing NetCDF file:
                   ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ... @^@^@
                   call from:  ROMS/Utility/inp_par.F, find_file
    
    Now, we get better information about the error:
    INP_PAR:FIND_FILE - empty file name string for standard input script
                        KeyWord: TIDENAME
    Found Error: 05   Line: 1396     Source: ROMS/Utility/inp_par.F
    
    READ_PHYPAR - Grid 01, could not find input file:  Oops unassigned file name.
                                                       Check standard input script...
    Found Error: 04   Line: 5674     Source: ROMS/Utility/read_phypar.F
    Found Error: 04   Line: 139      Source: ROMS/Utility/inp_par.F
    Found Error: 04   Line: 111      Source: ROMS/Drivers/nl_ocean.h
    
    This information is more useful and tell me that the file name for keyWord TIDENAME is missing from ocean.in and this input script needs to be updated to the lastest version.
  • Modified several routines, so the diagnostic printed to standard output have identical numbers in serial, serial with partitions, parallel distributed-memory (MPI), and parallel shared-memory (OpenMP) applications. Several of the global parallel reduction CRITICAL sections were updated.
Batch Modify
Note: See TracBatchModify for help on using batch modify.
Note: See TracQuery for help on using queries.