Tile Choice impacts Stability

Report or discuss software problems and other woes

Moderators: arango, robertson

Post Reply
Message
Author
jeffdorman
Posts: 1
Joined: Thu Dec 20, 2007 7:20 pm
Location: UC Berkeley

Tile Choice impacts Stability

#1 Unread post by jeffdorman »

Hi all,
I've run into a problem working with ROMS that has me a bit confused and I'm hopeful that someone out there can help.

I have set up a model run of the California Current and while testing the initialization of the model, it seems to runs fine on a single processor.

However, when I went to launch the run on 24 processors (tiling; NtileI == 4 NtileJ == 6) to run it for a longer period of time, the simulation "blows-up" within a few time steps.

To try and determine what was happening, I ran the model with various tiling. Some successful (NtileI x NtileJ; 1x1, 2x2, 3x3, 2x4, 2x5, 2x6) and some not (4x4, 5x5, 3x4, 4x3, 2x8). As a starting point, I am defining "successful" as running without blowing up for one day (480 time steps).

I am wondering if this has anything to do with the land mask on my grid. In certain places the mask extends up to 1/3rd of way along the I-dimension. So in certain tiling scenarios (e.g. 5x5) one or more tiles will be entirely masked. Could this be causing my runs to "blow-up" after just a few timesteps?

I created my grid in Seagrid and purposely tried to keep the grid simple (i.e. orthogonal) to begin with. I am aware that I could have reduced some of the masked area by creating a boundary that follows the general contour of the coastline, but was unsure of how non-orthogonality in the grid might influence model results.

I've copied and pasted the initial output from a representative model run below if that is helpful. Please let me know if there is any other information that might be useful.

Thanks for any input you have,

Jeff

******************************************************************************

Code: Select all

Model Input Parameters:  ROMS/TOMS version 3.2  
                          Thursday - February 26, 2009 -  9:20:04 AM
 -----------------------------------------------------------------------------

 Test of California Current System created by ROMSTOOLS

 Operating system : Linux
 CPU/hardware     : i686
 Compiler system  : pgi
 Compiler command : /opt/mpich/pgi/bin/mpif90
 Compiler flags   :  -fastsse -Mipa=fast -tp p6 -Mfree

 Input Script  : External/nptopc.in

 SVN Root URL  : https://www.myroms.org/svn/src/trunk
 SVN Revision  : 313M

 Local Root    : /home/jeffdorman2/nptopc_roms
 Header Dir    : /home/jeffdorman2/nptopc_roms/Projects/nptopc
 Header file   : nptopc.h
 Analytical Dir: /home/jeffdorman2/nptopc_roms/Projects/nptopc/Functionals

 Resolution, Grid 01: 0148x0183x020,  Parallel Nodes:   8,  Tiling: 002x004


 Physical Parameters, Grid: 01
 =============================

      87841  ntimes          Number of timesteps for 3-D equations.
    180.000  dt              Timestep size (s) for 3-D equations.
         30  ndtfast         Number of timesteps for 2-D equations between
                               each 3D timestep.
          1  ERstr           Starting ensemble/perturbation run number.
          1  ERend           Ending ensemble/perturbation run number.
          0  nrrec           Number of restart records to read from disk.
          T  LcycleRST       Switch to recycle time-records in restart file.
        120  nRST            Number of timesteps between the writing of data
                               into restart fields.
          1  ninfo           Number of timesteps between print of information
                               to standard output.
          T  ldefout         Switch to create a new output NetCDF file(s).
        120  nHIS            Number of timesteps between the writing fields
                               into history file.
          1  ntsAVG          Starting timestep for the accumulation of output
                               time-averaged data.
        120  nAVG            Number of timesteps between the writing of
                               time-averaged data into averages file.
          1  ntsDIA          Starting timestep for the accumulation of output
                               time-averaged diagnostics data.
        120  nDIA            Number of timesteps between the writing of
                               time-averaged data into diagnostics file.
 0.0000E+00  tnu2(01)        Horizontal, harmonic mixing coefficient (m2/s)
                               for tracer 01: temp
 0.0000E+00  tnu2(02)        Horizontal, harmonic mixing coefficient (m2/s)
                               for tracer 02: salt
 5.0000E+00  visc2           Horizontal, harmonic mixing coefficient (m2/s)
                               for momentum.
 1.0000E-06  Akt_bak(01)     Background vertical mixing coefficient (m2/s)
                               for tracer 01: temp
 1.0000E-06  Akt_bak(02)     Background vertical mixing coefficient (m2/s)
                               for tracer 02: salt
 1.0000E-05  Akv_bak         Background vertical mixing coefficient (m2/s)
                               for momentum.
 3.0000E-04  rdrg            Linear bottom drag coefficient (m/s).
 3.0000E-03  rdrg2           Quadratic bottom drag coefficient.
 2.0000E-02  Zob             Bottom roughness (m).
 1.0000E+01  blk_ZQ          Height (m) of surface air humidity measurement.
 1.0000E+01  blk_ZT          Height (m) of surface air temperature measurement.
 1.0000E+01  blk_ZW          Height (m) of surface winds measurement.
          1  lmd_Jwt         Jerlov water type.
 5.0000E+00  theta_s         S-coordinate surface control parameter.
 3.0000E-01  theta_b         S-coordinate bottom  control parameter.
     20.000  Tcline          S-coordinate surface/bottom layer width (m) used
                               in vertical coordinate stretching.
   1025.000  rho0            Mean density (kg/m3) for Boussinesq approximation.
      0.000  dstart          Time-stamp assigned to model initialization (days).
19000101.00  time_ref        Reference time for units attribute (yyyymmdd.dd)
 0.0000E+00  Tnudg(01)       Nudging/relaxation time scale (days)
                               for tracer 01: temp
 0.0000E+00  Tnudg(02)       Nudging/relaxation time scale (days)
                               for tracer 02: salt
 0.0000E+00  Znudg           Nudging/relaxation time scale (days)
                               for free-surface.
 0.0000E+00  M2nudg          Nudging/relaxation time scale (days)
                               for 2D momentum.
 0.0000E+00  M3nudg          Nudging/relaxation time scale (days)
                               for 3D momentum.
 0.0000E+00  obcfac          Factor between passive and active
                               open boundary conditions.
     14.000  T0              Background potential temperature (C) constant.
     35.000  S0              Background salinity (PSU) constant.
   1027.000  R0              Background density (kg/m3) used in linear Equation
                               of State.
 1.7000E-04  Tcoef           Thermal expansion coefficient (1/Celsius).
 0.0000E+00  Scoef           Saline contraction coefficient (1/PSU).
      1.000  gamma2          Slipperiness variable: free-slip (1.0) or 
                                                    no-slip (-1.0).
          T  Hout(idFsur)    Write out free-surface.
          T  Hout(idUbar)    Write out 2D U-momentum component.
          T  Hout(idVbar)    Write out 2D V-momentum component.
          T  Hout(idUvel)    Write out 3D U-momentum component.
          T  Hout(idVvel)    Write out 3D V-momentum component.
          T  Hout(idWvel)    Write out W-momentum component.
          T  Hout(idOvel)    Write out omega vertical velocity.
          T  Hout(idTvar)    Write out tracer 01: temp
          T  Hout(idTvar)    Write out tracer 02: salt
          T  Hout(idUsms)    Write out surface U-momentum stress.
          T  Hout(idVsms)    Write out surface V-momentum stress.
          T  Hout(idUbms)    Write out bottom U-momentum stress.
          T  Hout(idVbms)    Write out bottom V-momentum stress.
          T  Hout(idTsur)    Write out surface net heat flux.
          T  Hout(idTsur)    Write out surface net salt flux.
          T  Hout(idSrad)    Write out shortwave radiation flux.
          T  Hout(idLrad)    Write out longwave radiation flux.
          T  Hout(idLhea)    Write out latent heat flux.
          T  Hout(idShea)    Write out sensible heat flux.
          T  Hout(idevap)    Write out evaporation rate.
          T  Hout(idrain)    Write out rain rate.

 Output/Input Files:

             Output Restart File:  Data/night_090225runs/ocean_rst_24tile.nc
             Output History File:  Data/night_090225runs/ocean_his_24tile.nc
            Output Averages File:  Data/night_090225runs/ocean_avg_24tile.nc
         Output Diagnostics File:  Data/night_090225runs/ocean_dia_24tile.nc
                 Input Grid File:  External/roms_grd_NPtoPC.nc
    Input Nonlinear Initial File:  External/roms_ini_NPtoPC_ECCO_Y2004M7.nc
           Input Forcing File 01:  External/roms_blk_NCEP_jul2004_dec2005_NPtoPC_dlwr.nc
           Input Forcing File 02:  External/roms_blk_NCEP_jul2004_dec2005_NPtoPC_qair.nc
           Input Forcing File 03:  External/roms_blk_NCEP_jul2004_dec2005_NPtoPC_rain.nc
           Input Forcing File 04:  External/roms_blk_NCEP_jul2004_dec2005_NPtoPC_swra.nc
           Input Forcing File 05:  External/roms_blk_NCEP_jul2004_dec2005_NPtoPC_tair.nc
           Input Forcing File 06:  External/roms_blk_NCEP_jul2004_dec2005_NPtoPC_uwin.nc
           Input Forcing File 07:  External/roms_blk_NCEP_jul2004_dec2005_NPtoPC_vwin.nc
             Input Boundary File:  External/roms_bry_NPtoPC_ECCO_jul2004_dec2005_npzdeqzeros.nc
         Biology Parameters File:  External/npzd_Powell.in

 Tile partition information for Grid 01:  0148x0183x0020  tiling: 002x004

     tile     Istr     Iend     Jstr     Jend     Npts

        0        1       74        1       46    68080
        1       75      148        1       46    68080
        2        1       74       47       92    68080
        3       75      148       47       92    68080
        4        1       74       93      138    68080
        5       75      148       93      138    68080
        6        1       74      139      183    66600
        7       75      148      139      183    66600

 Tile minimum and maximum fractional grid coordinates:
   (interior points only)

     tile     Xmin     Xmax     Ymin     Ymax     grid

        0     0.50    74.50     0.50    46.50  RHO-points
        1    74.50   148.50     0.50    46.50  RHO-points
        2     0.50    74.50    46.50    92.50  RHO-points
        3    74.50   148.50    46.50    92.50  RHO-points
        4     0.50    74.50    92.50   138.50  RHO-points
        5    74.50   148.50    92.50   138.50  RHO-points
        6     0.50    74.50   138.50   183.50  RHO-points
        7    74.50   148.50   138.50   183.50  RHO-points

        0     1.00    74.50     0.50    46.50    U-points
        1    74.50   148.00     0.50    46.50    U-points
        2     1.00    74.50    46.50    92.50    U-points
        3    74.50   148.00    46.50    92.50    U-points
        4     1.00    74.50    92.50   138.50    U-points
        5    74.50   148.00    92.50   138.50    U-points
        6     1.00    74.50   138.50   183.50    U-points
        7    74.50   148.00   138.50   183.50    U-points

        0     0.50    74.50     1.00    46.50    V-points
        1    74.50   148.50     1.00    46.50    V-points
        2     0.50    74.50    46.50    92.50    V-points
        3    74.50   148.50    46.50    92.50    V-points
        4     0.50    74.50    92.50   138.50    V-points
        5    74.50   148.50    92.50   138.50    V-points
        6     0.50    74.50   138.50   183.00    V-points
        7    74.50   148.50   138.50   183.00    V-points

 Maximum halo size in XI and ETA directions:

               HaloSizeI(1) =     172
               HaloSizeJ(1) =     116
                TileSide(1) =      80
                TileSize(1) =    4160



 NPZD Model Parameters, Grid: 01
 ===============================

          1  BioIter         Number of iterations for nonlinear convergence.
 1.7000E+01  BioIni(iNO3_)   Nitrate initial concentration (mmol/m3).
 1.0000E+00  BioIni(iPhyt)   Phytoplankton initial concentration (mmol/m3).
 1.0000E+00  BioIni(iZoop)   Zooplankton initial concentration (mmol/m3).
 1.0000E+00  BioIni(iSDet)   Small detritus initial concentration (mmol/m3).
 4.3000E-01  PARfrac         Fraction of shortwave radiation that is
                               photosynthetically active (nondimensional).
 6.7000E-02  AttSW           Light attenuation of seawater (m-1).
 9.5000E-03  AttPhy          Light attenuation by phytoplankton (m2/mmole_N).
 2.5000E-02  PhyIS           Phytoplankton growth, initial slope of P-I curve
                               (m2/W).
 1.5000E+00  Vm_NO3          Nitrate upatake rate (day-1).
 1.0000E-01  PhyMRD          Phytoplankton mortality rate to Detritus (day-1)
 0.0000E+00  PhyMRN          Phytoplankton mortality rate to Nitrogen (day-1)
 1.0000E+00  K_NO3           Inverse half-saturation for phytoplankton NO3
                               uptake (1/(mmol m-3)).
 6.0000E-02  Ivlev           Ivlev constant for zooplankton grazing
                               (nondimensional).
 5.2000E-01  ZooGR           Zooplankton maximum growth rate (day-1).
 0.0000E+00  ZooEED          Zooplankton excretion efficiency to Detritus
                               pool (nondimensional).
 3.0000E-01  ZooEEN          Zooplankton excretion efficiency to Nitrogen
                               pool (nondimensional).
 1.4500E-01  ZooMRD          Zooplankton mortality rate to Detritus (day-1).
 0.0000E+00  ZooMRN          Zooplankton mortality rate to Nitrogen (day-1).
 1.0300E+00  DetRR           Detritus remineralization rate (day-1).
 0.0000E+00  wPhy            Phytoplankton sinking rate (m/day).
 8.0000E+00  wDet            Detrital sinking rate (m/day).
 0.0000E+00  tnu2(03)        Horizontal, harmonic mixing coefficient (m2/s)
                               for tracer 03: NO3
 0.0000E+00  tnu2(04)        Horizontal, harmonic mixing coefficient (m2/s)
                               for tracer 04: phytoplankton
 0.0000E+00  tnu2(05)        Horizontal, harmonic mixing coefficient (m2/s)
                               for tracer 05: zooplankton
 0.0000E+00  tnu2(06)        Horizontal, harmonic mixing coefficient (m2/s)
                               for tracer 06: detritus
 5.0000E-06  Akt_bak(03)     Background vertical mixing coefficient (m2/s)
                               for tracer 03: NO3
 5.0000E-06  Akt_bak(04)     Background vertical mixing coefficient (m2/s)
                               for tracer 04: phytoplankton
 5.0000E-06  Akt_bak(05)     Background vertical mixing coefficient (m2/s)
                               for tracer 05: zooplankton
 5.0000E-06  Akt_bak(06)     Background vertical mixing coefficient (m2/s)
                               for tracer 06: detritus
 0.0000E+00  Tnudg(03)       Nudging/relaxation time scale (days)
                               for tracer 03: NO3
 0.0000E+00  Tnudg(04)       Nudging/relaxation time scale (days)
                               for tracer 04: phytoplankton
 0.0000E+00  Tnudg(05)       Nudging/relaxation time scale (days)
                               for tracer 05: zooplankton
 0.0000E+00  Tnudg(06)       Nudging/relaxation time scale (days)
                               for tracer 06: detritus
          T  Hout(idTvar)    Write out tracer 03: NO3
          T  Hout(idTvar)    Write out tracer 04: phytoplankton
          T  Hout(idTvar)    Write out tracer 05: zooplankton
          T  Hout(idTvar)    Write out tracer 06: detritus

 Activated C-preprocessing Options:

  NPTOPC             Test of California Current System created by ROMSTOOLS
  ATM_PRESS          Impose atmospheric pressure onto sea surface.
  ANA_BIOLOGY        Analytical biology initial conditions.
  ANA_BPFLUX         Analytical bottom passive tracers fluxes.
  ANA_BSFLUX         Analytical kinematic bottom salinity flux.
  ANA_BTFLUX         Analytical kinematic bottom temperature flux.
  ANA_PAIR           Analytical surface air pressure.
  ANA_SPFLUX         Analytical surface passive tracer fluxes.
  ASSUMED_SHAPE      Using assumed-shape arrays.
  AVERAGES           Writing out time-averaged fields.
  BULK_FLUXES        Surface bulk fluxes parametererization.
  DIAGNOSTICS_TS     Computing and writing tracer diagnostic terms.
  DIAGNOSTICS_UV     Computing and writing momentum diagnostic terms.
  DJ_GRADPS          Parabolic Splines density Jacobian (Shchepetkin, 2002).
  DOUBLE_PRECISION   Double precision arithmetic.
  EASTERN_WALL       Wall boundary at Eastern edge.
  EMINUSP            Compute Salt Flux using E-P.
  LMD_CONVEC         LMD convective mixing due to shear instability.
  LMD_MIXING         Large/McWilliams/Doney interior mixing.
  LMD_NONLOCAL       LMD convective nonlocal transport.
  LMD_RIMIX          LMD diffusivity due to shear instability.
  LMD_SKPP           KPP surface boundary layer mixing.
  LONGWAVE_OUT       Compute outgoing longwave radiation internally.
  MASKING            Land/Sea masking.
  MIX_GEO_TS         Mixing of tracers along geopotential surfaces.
  MIX_GEO_UV         Mixing of momentum along geopotential surfaces.
  MPI                MPI distributed-memory configuration.
  NONLINEAR          Nonlinear Model.
  !NONLIN_EOS        Linear Equation of State for seawater.
  NORTH_FSCHAPMAN    Northern edge, free-surface, Chapman condition.
  NORTH_M2FLATHER    Northern edge, 2D momentum, Flather condition.
  NORTH_M3CLAMPED    Northern edge, 3D momentum, Clamped condition.
  NORTH_TCLAMPED     Northern edge, tracers, Clamped condition.
  NPZD_POWELL        NPZD Biological Model, Powell et al. fomulation.
  POWER_LAW          Power-law shape time-averaging barotropic filter.
  PROFILE            Time profiling activated .
  !RST_SINGLE        Double precision fields in restart NetCDF file.
  SALINITY           Using salinity.
  SOLAR_SOURCE       Solar Radiation Source Term.
  SOLVE3D            Solving 3D Primitive Equations.
  SOUTH_FSCHAPMAN    Southern edge, free-surface, Chapman condition.
  SOUTH_M2FLATHER    Southern edge, 2D momentum, Flather condition.
  SOUTH_M3CLAMPED    Southern edge, 3D momentum, Clamped condition.
  SOUTH_TCLAMPED     Southern edge, tracers, Clamped condition.
  SPLINES            Conservative parabolic spline reconstruction.
  TS_U3HADVECTION    Third-order upstream horizontal advection of tracers.
  TS_C4VADVECTION    Fourth-order centered vertical advection of tracers.
  TS_DIF2            Harmonic mixing of tracers.
  UV_ADV             Advection of momentum.
  UV_COR             Coriolis term.
  UV_U3HADVECTION    Third-order upstream horizontal advection of 3D momentum.
  UV_C4VADVECTION    Fourth-order centered vertical advection of momentum.
  UV_LDRAG           Linear bottom stress.
  UV_VIS2            Harmonic mixing of momentum.
  VAR_RHO_2D         Variable density barotropic mode.
  WEST_FSCHAPMAN     Western edge, free-surface, Chapman condition.
  WEST_M2FLATHER     Western edge, 2D momentum, Flather condition.
  WEST_M3CLAMPED     Western edge, 3D momentum, Clamped condition.
  WEST_TCLAMPED      Western edge, tracers, Clamped condition.

 INITIAL: Configuring and initializing forward nonlinear model ...


 Vertical S-coordinate System: 

 level   S-coord     Cs-curve          at_hmin  over_slope     at_hmax

    20   0.0000000   0.0000000           0.000       0.000       0.000
    19  -0.0500000  -0.0036887          -1.000      -9.633     -18.266
    18  -0.1000000  -0.0083498          -2.000     -21.541     -41.083
    17  -0.1500000  -0.0146352          -3.000     -37.252     -71.504
    16  -0.2000000  -0.0234720          -4.000     -58.933    -113.866
    15  -0.2500000  -0.0361429          -5.000     -89.587    -174.175
    14  -0.3000000  -0.0542976          -6.000    -133.076    -260.152
    13  -0.3500000  -0.0797586          -7.000    -193.664    -380.328
    12  -0.4000000  -0.1139562          -8.000    -274.698    -541.397
    11  -0.4500000  -0.1570182          -9.000    -376.479    -743.958
    10  -0.5000000  -0.2070749         -10.000    -494.630    -979.260
     9  -0.5500000  -0.2607175         -11.000    -621.173   -1231.346
     8  -0.6000000  -0.3147622         -12.000    -748.657   -1485.314
     7  -0.6500000  -0.3680293         -13.000    -874.321   -1735.642
     6  -0.7000000  -0.4218448         -14.000   -1001.269   -1988.538
     5  -0.7500000  -0.4794204         -15.000   -1137.016   -2259.033
     4  -0.8000000  -0.5450552         -16.000   -1291.625   -2567.251
     3  -0.8500000  -0.6237264         -17.000   -1476.744   -2936.489
     2  -0.9000000  -0.7211044         -18.000   -1705.644   -3393.288
     1  -0.9500000  -0.8438388         -19.000   -1993.887   -3968.773
     0  -1.0000000  -1.0000000         -20.000   -2360.360   -4700.720

 Time Splitting Weights: ndtfast =  30    nfast =  42

    Primary            Secondary            Accumulated to Current Step

  1-0.0008094437383769 0.0333333333333333-0.0008094437383769 0.0333333333333333
  2-0.0014053566728197 0.0333603147912792-0.0022148004111966 0.0666936481246126
  3-0.0017877524645903 0.0334071600137066-0.0040025528757869 0.1001008081383191
  4-0.0019566842408176 0.0334667517625262-0.0059592371166045 0.1335675599008453
  5-0.0019122901320372 0.0335319745705535-0.0078715272486418 0.1670995344713988
  6-0.0016548570247459 0.0335957175749547-0.0095263842733877 0.2006952520463535
  7-0.0011849025289723 0.0336508794757796-0.0107112868023600 0.2343461315221331
  8-0.0005032751608631 0.0336903762267453-0.0112145619632231 0.2680365077488784
  9 0.0003887272597151 0.0337071520654408-0.0108258347035081 0.3017436598143192
 10 0.0014892209965583 0.0336941944901169-0.0093366137069498 0.3354378543044362
 11 0.0027955815694920 0.0336445537902317-0.0065410321374577 0.3690824080946679
 12 0.0043042707117221 0.0335513677379153-0.0022367614257356 0.4026337758325831
 13 0.0060106451121704 0.0334078920475245 0.0037738836864348 0.4360416678801077
 14 0.0079087469427945 0.0332075372104522 0.0116826306292293 0.4692492050905599
 15 0.0099910761708919 0.0329439123123590 0.0216737068001213 0.5021931174029188
 16 0.0122483446563884 0.0326108764399960 0.0339220514565096 0.5348039938429148
 17 0.0146692120341107 0.0322025982847830 0.0485912634906204 0.5670065921276978
 18 0.0172400033810439 0.0317136245503127 0.0658312668716642 0.5987202166780104
 19 0.0199444086685725 0.0311389577709445 0.0857756755402367 0.6298591744489550
 20 0.0227631639997064 0.0304741441486588 0.1085388395399431 0.6603333185976138
 21 0.0256737146312910 0.0297153720153352 0.1342125541712341 0.6900486906129490
 22 0.0286498597812016 0.0288595815276255 0.1628624139524357 0.7189082721405747
 23 0.0316613792205219 0.0279045862015855 0.1945237931729576 0.7468128583421601
 24 0.0346736416507075 0.0268492068942347 0.2291974348236651 0.7736620652363948
 25 0.0376471948657328 0.0256934188392112 0.2668446296893979 0.7993554840756060
 26 0.0405373376992232 0.0244385123436867 0.3073819673886211 0.8237939964192927
 27 0.0432936737565710 0.0230872677537126 0.3506756411451921 0.8468812641730054
 28 0.0458596469320356 0.0216441452951603 0.3965352880772277 0.8685254094681657
 29 0.0481720587108284 0.0201154903974257 0.4447073467880561 0.8886408998655913
 30 0.0501605672561819 0.0185097551070648 0.4948679140442381 0.9071506549726561
 31 0.0517471682814030 0.0168377361985254 0.5466150823256410 0.9239883911711814
 32 0.0528456577069106 0.0151128305891453 0.5994607400325517 0.9391012217603267
 33 0.0533610761022577 0.0133513086655816 0.6528218161348094 0.9524525304259084
 34 0.0531891349131380 0.0115726061288397 0.7060109510479473 0.9640251365547481
 35 0.0522156244733761 0.0097996349650684 0.7582265755213234 0.9738247715198165
 36 0.0503158038019031 0.0080591141492892 0.8085423793232264 0.9818838856691057
 37 0.0473537721847154 0.0063819206892258 0.8558961515079417 0.9882658063583316
 38 0.0431818225418188 0.0048034616164019 0.8990779740497604 0.9930692679747334
 39 0.0376397765791564 0.0033640675316746 0.9367177506289169 0.9964333355064081
 40 0.0305543017255207 0.0021094083123694 0.9672720523544376 0.9985427438187775
 41 0.0217382098544505 0.0010909315881854 0.9890102622088881 0.9996336754069629
 42 0.0109897377911119 0.0003663245930371 0.9999999999999998 1.0000000000000000

 ndtfast, nfast =   30  42   nfast/ndtfast = 1.40000

 Centers of gravity and integrals (values must be 1, 1, approx 1/2, 1, 1):

    1.000000000000 1.047601458608 0.523800729304 1.000000000000 1.000000000000

 Power filter parameters, Fgamma, gamma =  0.28400   0.18933

 Minimum X-grid spacing, DXmin =  2.37987620E+00 km
 Maximum X-grid spacing, DXmax =  3.21666520E+00 km
 Minimum Y-grid spacing, DYmin =  4.18668041E+00 km
 Maximum Y-grid spacing, DYmax =  6.65924384E+00 km
 Minimum Z-grid spacing, DZmin =  1.00000000E+00 m
 Maximum Z-grid spacing, DZmax =  7.31946956E+02 m

 Minimum barotropic Courant Number =  2.97227410E-02
 Maximum barotropic Courant Number =  4.97801292E-01
 Maximum Coriolis   Courant Number =  1.84785633E-02

User avatar
arango
Site Admin
Posts: 1360
Joined: Wed Feb 26, 2003 4:41 pm
Location: DMCS, Rutgers University
Contact:

Re: Tile Choice impacts Stability

#2 Unread post by arango »

I don't know what is your problem. However, I don't really understand why do you want to run a 148x183x20 in so many parallel nodes? This is a relatively small grid; using that many nodes is an overkill. The more nodes that you use the more MPI communications that are needed and the larger the overhead :!: I am sure that such application will run faster in less number of processes. If you have that many CPU available to use, I recommend to run different set-ups simultaneously. That will be a better use of the computational resources. You need to increase the number of nodes as you problem becomes larger and larger. Our benchmark 3 is 2048x256x30 and we use around 24 to 32 nodes.

I have say many times in this forum that there is an optimal number of nodes to use in a particular grid. Users need to benchmark which partition combination is faster before submitting large production runs.

User avatar
kate
Posts: 4091
Joined: Wed Jul 02, 2003 5:29 pm
Location: CFOS/UAF, USA

Re: Tile Choice impacts Stability

#3 Unread post by kate »

You don't say how it blows up. I would try a shorter timestep, which may or may not make it more stable. I would plot the restart field at the time it's blowing up - can you see which field went bad and where? I love, love, love that ROMS saves a restart snapshot before things get all the way to NaN and I added a similar check for the bio variables to my diag.F.

Post Reply