How to solve the speed trouble on the open boundaries?

Report or discuss software problems and other woes

Moderators: arango, robertson

Post Reply
Message
Author
yuke4500

How to solve the speed trouble on the open boundaries?

#1 Unread post by yuke4500 »

Hi, everyone
I have a DIAG speed trouble when I tried to run a long-period (~20 years) WRF-ROMS coupling simulation. But my simulations always crashed after about 6~7 years. I tried to change the time step, the relaxation time scales, with or without the tide forcing... All of these could not solve the speed trouble.
ncview.temp.pdf
(46.64 KiB) Downloaded 178 times
ncview.zeta.pdf
(33.59 KiB) Downloaded 161 times
ncview.u.pdf
(46.52 KiB) Downloaded 168 times
ncview.v.pdf
(40.02 KiB) Downloaded 166 times
The range of v is -6~28m/s and the picture only displays -6~8m/s.
This is my latest cpp file,

#define ROMS_MODEL
#define WRF_MODEL
#define MCT_LIB
#define MCT_INTERP_OC2AT

#define WET_DRY

# define UV_ADV
# define UV_COR
# define UV_VIS2
# define UV_QDRAG
# define MIX_S_UV
# undef TS_FIXED
# define TS_U3HADVECTION
# define TS_C4VADVECTION
# undef TS_MPDATA

# define DJ_GRADPS
# define TS_DIF2
# define MIX_GEO_TS

# define CURVGRID

# define SALINITY
# define SOLVE3D
# define NONLIN_EOS

/* Grid and Initial */
# define MASKING

# undef SPLINES

# define LONGWAVE
/*# define SHORTWAVE*/
# undef BULK_FLUXES
# define ATM2OCN_FLUXES
# undef ANA_SSFLUX
# undef LONGWAVE_OUT

# define ATM_PRESS
# define ANA_BTFLUX
# define ANA_BSFLUX

# define EMINUSP
# define SOLAR_SOURCE

/*
# define SSH_TIDES
# define UV_TIDES
# define RAMP_TIDES
# define ADD_FSOBC
# define ADD_M2OBC
*/

#define LMD_MIXING
#ifdef LMD_MIXING
# define LMD_RIMIX
# define LMD_CONVEC
# define LMD_SKPP
# define LMD_NONLOCAL
#endif

/*
OPTIONS for the Generic Length-Scale closure (Warner et al., 2005)
# define KANTHA_CLAYSON
# define CHARNOK
# define CRAIG_BANNER
# define N2S2_HORAVG
*/

/* Output */
# undef DIAGNOSTICS_BIO /*use if writing out biological diagnostics*/
# define DIAGNOSTICS_UV
# define DIAGNOSTICS_TS
/* Model configuration OPTIONS: */

# define AVERAGES /*use if writing out NLM time-averaged data*/
# undef AVERAGES2 /*use if writing out secondary time-averaged data*/
# undef HISTORY2 /*use if writing out secondary history data*/
# undef AVERAGES_DETIDE /*use if writing out NLM time-averaged detided fields*/
# undef AD_AVERAGES /*use if writing out ADM time-averaged data*/
# undef RP_AVERAGES /*use if writing out TLM time-averaged data*/
# undef TL_AVERAGES /*use if writing out ADM time-averaged data*/


/*
** Added defs
*/

#define UV_U3HADVECTION
#define UV_C4VADVECTION
#define VAR_RHO_2D

#define VISC_GRID

#define RADIATION_2D
#define LIMIT_BSTRESS

#define PERFECT_RESTART
#define DEFLATE
#define HDF5

Part of the log file:
Operating system : Linux
CPU/hardware : x86_64
Compiler system : ifort
Compiler command : /opt/intel/intel2015u6/impi/5.0.3.049/intel64/bin/mpiifort
Compiler flags : -fp-model precise -ip -O3 -traceback -check uninit -ip -O3 -I/gpfs2/keyu/libs/MCT/MCT-2.6.0-intel2015u6-icc-ifort-mpiifort/include -I/gpfs2/keyu/work/models/CPL/COAWST/COAWST-EACLI-test2//WRF/main -I/gpfs2/keyu/work/models/CPL/COAWST/COAWST-EACLI-test2//WRF/external/esmf_time_f90 -I/gpfs2/keyu/work/models/CPL/COAWST/COAWST-EACLI-test2//WRF/frame -I/gpfs2/keyu/work/models/CPL/COAWST/COAWST-EACLI-test2//WRF/share
MPI Communicator : -2080374784 PET size = 112

Input Script :

SVN Root URL :
SVN Revision :

Local Root : /gpfs2/keyu/work/models/CPL/COAWST/COAWST-EACLI-test2
Header Dir : /gpfs2/keyu/work/models/CPL/COAWST/COAWST-EACLI-test2/Projects/CORDEXEA
Header file : cordexea.h
Analytical Dir : /gpfs2/keyu/work/models/CPL/COAWST/COAWST-EACLI-test2/Projects/CORDEXEA

Resolution, Grid 01: 478x313x41, Parallel Nodes: 112, Tiling: 14x8


Physical Parameters, Grid: 01
=============================

1051920 ntimes Number of timesteps for 3-D equations.
120.000 dt Timestep size (s) for 3-D equations.
30 ndtfast Number of timesteps for 2-D equations between
each 3D timestep.
1 ERstr Starting ensemble/perturbation run number.
1 ERend Ending ensemble/perturbation run number.
-1 nrrec Number of restart records to read from disk.
T LcycleRST Switch to recycle time-records in restart file.
720 nRST Number of timesteps between the writing of data
into restart fields.
1 ninfo Number of timesteps between print of information
to standard output.
T ldefout Switch to create a new output NetCDF file(s).
180 nHIS Number of timesteps between the writing fields
into history file.
14400 ndefHIS Number of timesteps between creation of new
history files.
0 nQCK Number of timesteps between the writing fields
into quicksave file.
1 ntsAVG Starting timestep for the accumulation of output
time-averaged data.
720 nAVG Number of timesteps between the writing of
time-averaged data into averages file.
14400 ndefAVG Number of timesteps between creation of new
time-averaged file.
1 ntsDIA Starting timestep for the accumulation of output
time-averaged diagnostics data.
720 nDIA Number of timesteps between the writing of
time-averaged data into diagnostics file.
14400 ndefDIA Number of timesteps between creation of new
diagnostic file.
2.0000E+00 nl_tnu2(01) NLM Horizontal, harmonic mixing coefficient
(m2/s) for tracer 01: temp
2.0000E+00 nl_tnu2(02) NLM Horizontal, harmonic mixing coefficient
(m2/s) for tracer 02: salt
2.0000E+00 nl_visc2 NLM Horizontal, harmonic mixing coefficient
(m2/s) for momentum.
F LuvSponge Turning OFF sponge on horizontal momentum.
F LtracerSponge(01) Turning OFF sponge on tracer 01: temp
F LtracerSponge(02) Turning OFF sponge on tracer 02: salt
1.0000E-06 Akt_bak(01) Background vertical mixing coefficient (m2/s)
for tracer 01: temp
1.0000E-06 Akt_bak(02) Background vertical mixing coefficient (m2/s)
for tracer 02: salt
1.0000E-05 Akv_bak Background vertical mixing coefficient (m2/s)
for momentum.
3.0000E-04 rdrg Linear bottom drag coefficient (m/s).
2.5000E-02 rdrg2 Quadratic bottom drag coefficient.
2.0000E-02 Zob Bottom roughness (m).
1.0000E-01 Dcrit Minimum depth for wetting and drying (m).
1 lmd_Jwt Jerlov water type.
2 Vtransform S-coordinate transformation equation.
4 Vstretching S-coordinate stretching function.
9.0000E+00 theta_s S-coordinate surface control parameter.
0.0000E+00 theta_b S-coordinate bottom control parameter.
50.000 Tcline S-coordinate surface/bottom layer width (m) used
in vertical coordinate stretching.
1025.000 rho0 Mean density (kg/m3) for Boussinesq approximation.
0.000 dstart Time-stamp assigned to model initialization (days).
19880101.00 time_ref Reference time for units attribute (yyyymmdd.dd)
5.0000E+00 Tnudg(01) Nudging/relaxation time scale (days)
for tracer 01: temp
5.0000E+00 Tnudg(02) Nudging/relaxation time scale (days)
for tracer 02: salt
0.0000E+00 Tnudg_SSS Nudging/relaxation time scale (days)
for sea surface salinity.
0.0000E+00 Znudg Nudging/relaxation time scale (days)
for free-surface.
0.0000E+00 M2nudg Nudging/relaxation time scale (days)
for 2D momentum.
5.0000E+00 M3nudg Nudging/relaxation time scale (days)
for 3D momentum.
1.0000E+01 obcfac Factor between passive and active
open boundary conditions.
F VolCons(1) NLM western edge boundary volume conservation.
F VolCons(2) NLM southern edge boundary volume conservation.
F VolCons(3) NLM eastern edge boundary volume conservation.
F VolCons(4) NLM northern edge boundary volume conservation.
10.000 T0 Background potential temperature (C) constant.
30.000 S0 Background salinity (PSU) constant.
-1.000 gamma2 Slipperiness variable: free-slip (1.0) or
no-slip (-1.0).
F LuvSrc Turning OFF momentum point Sources/Sinks.
F LwSrc Turning OFF volume influx point Sources/Sinks.
F LtracerSrc(01) Turning OFF point Sources/Sinks on tracer 01: temp
F LtracerSrc(02) Turning OFF point Sources/Sinks on tracer 02: salt
F LsshCLM Turning OFF processing of SSH climatology.
F Lm2CLM Turning OFF processing of 2D momentum climatology.
F Lm3CLM Turning OFF processing of 3D momentum climatology.
F LtracerCLM(01) Turning OFF processing of climatology tracer 01: temp
F LtracerCLM(02) Turning OFF processing of climatology tracer 02: salt
F LnudgeM2CLM Turning OFF nudging of 2D momentum climatology.
F LnudgeM3CLM Turning OFF nudging of 3D momentum climatology.
F LnudgeTCLM(01) Turning OFF nudging of climatology tracer 01: temp
F LnudgeTCLM(02) Turning OFF nudging of climatology tracer 02: salt

T Hout(idFsur) Write out free-surface.
T Hout(idUbar) Write out 2D U-momentum component.
T Hout(idVbar) Write out 2D V-momentum component.
T Hout(idUvel) Write out 3D U-momentum component.
T Hout(idVvel) Write out 3D V-momentum component.
T Hout(idWvel) Write out W-momentum component.
T Hout(idOvel) Write out omega vertical velocity.
T Hout(idTvar) Write out tracer 01: temp
T Hout(idTvar) Write out tracer 02: salt
T Hout(idVvis) Write out vertical viscosity: AKv.
T Hout(idTdif) Write out vertical diffusion: AKt(itemp).
T Hout(idHsbl) Write out depth of surface boundary layer.

T Aout(idZZav) Write out averaged quadratic <zeta*zeta> term.
T Aout(idU2av) Write out averaged quadratic <ubar*ubar> term.
T Aout(idV2av) Write out averaged quadratic <vbar*vbar> term.
T Aout(idHUav) Write out averaged u-volume flux, Huon.
T Aout(idHVav) Write out averaged v-volume flux, Hvom.
T Aout(idUUav) Write out averaged quadratic <u*u> term.
T Aout(idUVav) Write out averaged quadratic <u*v> term.
T Aout(idVVav) Write out averaged quadratic <v*v> term.
T Aout(idTTav) Write out averaged <t*t> for tracer 01: temp
T Aout(idTTav) Write out averaged <t*t> for tracer 02: salt
T Aout(idUTav) Write out averaged <u*t> for tracer 01: temp
T Aout(idUTav) Write out averaged <u*t> for tracer 02: salt
T Aout(idVTav) Write out averaged <v*t> for tracer 01: temp
T Aout(idVTav) Write out averaged <v*t> for tracer 02: salt
T Aout(iHUTav) Write out averaged <Huon*t> for tracer 01: temp
T Aout(iHUTav) Write out averaged <Huon*t> for tracer 02: salt
T Aout(iHVTav) Write out averaged <Hvom*t> for tracer 01: temp
T Aout(iHVTav) Write out averaged <Hvom*t> for tracer 02: salt


T Dout(M3rate) Write out 3D momentum acceleration.
T Dout(M3pgrd) Write out 3D momentum pressure gradient.
T Dout(M3fcor) Write out 3D momentum Coriolis force.
T Dout(M3hadv) Write out 3D momentum horizontal advection.
T Dout(M3xadv) Write out 3D momentum horizontal X-advection.
T Dout(M3yadv) Write out 3D momentum horizontal Y-advection.
T Dout(M3vadv) Write out 3D momentum vertical advection.
T Dout(M3hvis) Write out 3D momentum horizontal viscosity.
T Dout(M3xvis) Write out 3D momentum horizontal X-viscosity.
T Dout(M3yvis) Write out 3D momentum horizontal Y-viscosity.
T Dout(M3vvis) Write out 3D momentum vertical viscosity.

T Dout(iTrate) Write out rate of change of tracer 01: temp
T Dout(iTrate) Write out rate of change of tracer 02: salt
T Dout(iThadv) Write out horizontal advection, tracer 01: temp
T Dout(iThadv) Write out horizontal advection, tracer 02: salt
T Dout(iTxadv) Write out horizontal X-advection, tracer 01: temp
T Dout(iTxadv) Write out horizontal X-advection, tracer 02: salt
T Dout(iTyadv) Write out horizontal Y-advection, tracer 01: temp
T Dout(iTyadv) Write out horizontal Y-advection, tracer 02: salt
T Dout(iTvadv) Write out vertical advection, tracer 01: temp
T Dout(iTvadv) Write out vertical advection, tracer 02: salt
T Dout(iThdif) Write out horizontal diffusion, tracer 01: temp
T Dout(iThdif) Write out horizontal diffusion, tracer 02: salt
T Dout(iTxdif) Write out horizontal X-diffusion, tracer 01: temp
T Dout(iTxdif) Write out horizontal X-diffusion, tracer 02: salt
T Dout(iTydif) Write out horizontal Y-diffusion , tracer 01: temp
T Dout(iTydif) Write out horizontal Y-diffusion , tracer 02: salt
T Dout(iTsdif) Write out horizontal S-diffusion, tracer 01: temp
T Dout(iTsdif) Write out horizontal S-diffusion, tracer 02: salt
T Dout(iTvdif) Write out vertical diffusion, tracer 01: temp
T Dout(iTvdif) Write out vertical diffusion, tracer 02: salt
1 shuffle NetCDF-4/HDF5 file format shuffle filer flag.
1 deflate NetCDF-4/HDF5 file format deflate filer flag.
1 deflate_level NetCDF-4/HDF5 file format deflate level parameter.

Output/Input Files:

Output Restart File: ocnoutput/ocean_rst.nc
Prefix for History Files: ocnoutput/ocean_his
Prefix for Averages Files: ocnoutput/ocean_avg



The lines at the end of the log file:
1548476 1993-11-20 15:52:00.00 6.210746E-02 2.169523E+04 2.169530E+04 1.109066E+17
(463,001,27) 3.444431E-03 2.172299E-02 4.438351E-01 1.746114E+01
Timing for main: time 1993-11-20_17:16:30 on domain 1: 0.10999 elapsed seconds
Timing for main: time 1993-11-20_17:16:30 on domain 1: 0.10999 elapsed seconds
Timing for main: time 1993-11-20_17:18:00 on domain 1: 0.11461 elapsed seconds
Timing for main: time 1993-11-20_17:18:00 on domain 1: 0.11461 elapsed seconds
1548477 1993-11-20 15:54:00.00 6.218470E-02 2.169523E+04 2.169529E+04 1.109066E+17
(463,001,26) 2.245054E-03 2.137597E-02 4.517182E-01 1.839780E+01
Timing for main: time 1993-11-20_17:19:30 on domain 1: 0.11724 elapsed seconds
Timing for main: time 1993-11-20_17:19:30 on domain 1: 0.11724 elapsed seconds
Timing for main: time 1993-11-20_17:21:00 on domain 1: 0.10964 elapsed seconds
Timing for main: time 1993-11-20_17:21:00 on domain 1: 0.10964 elapsed seconds
1548478 1993-11-20 15:56:00.00 6.226221E-02 2.169522E+04 2.169528E+04 1.109065E+17
(463,001,26) 1.993098E-03 2.126670E-02 4.585752E-01 1.927556E+01
Timing for main: time 1993-11-20_17:22:30 on domain 1: 0.10917 elapsed seconds
Timing for main: time 1993-11-20_17:22:30 on domain 1: 0.10917 elapsed seconds
Timing for main: time 1993-11-20_17:24:00 on domain 1: 0.11278 elapsed seconds
Timing for main: time 1993-11-20_17:24:00 on domain 1: 0.11278 elapsed seconds
1548479 1993-11-20 15:58:00.00 6.234014E-02 2.169521E+04 2.169527E+04 1.109065E+17
(463,001,26) 1.802130E-03 2.111222E-02 4.651481E-01 2.006906E+01
DIAG speed trouble 20.0690640433859
Found Error: 01 Line: 351 Source: ROMS/Nonlinear/main3d.F
Found Error: 01 Line: 320 Source: ROMS/Drivers/nl_ocean.h

Blowing-up: Saving latest model state into RESTART file

DIAG speed ijk 28.2925733849489 290 2 16
1
Timing for main: time 1993-11-20_17:25:30 on domain 1: 0.10901 elapsed seconds


I used the soda-3.7.2 5-day as the initial condition and open boundary forcing.
The nudging time scales:
TNUDG == 5.0d0 5.0d0 ! days
ZNUDG == 0.0d0 ! days
M2NUDG == 0.0d0 ! days
M3NUDG == 5.0d0 ! days

and OBCFAC == 10.0d0


The *.in and coupling*.in file are are also attached.
cordexea.in
(120.29 KiB) Downloaded 169 times
coupling_cordexea.in
(6.91 KiB) Downloaded 154 times
Could anyone tell me how to solve this problem?
Regards,
Yu

User avatar
kate
Posts: 4088
Joined: Wed Jul 02, 2003 5:29 pm
Location: CFOS/UAF, USA

Re: How to solve the speed trouble on the open boundaries?

#2 Unread post by kate »

I typically add a nudging to a monthly climatology in a band around the edge of the domain, ramping to zero in the middle. Note that this will overwrite TNUDG and friends in the default ROMS code.

yuke4500

Re: How to solve the speed trouble on the open boundaries?

#3 Unread post by yuke4500 »

kate wrote: Sun Apr 04, 2021 5:18 pm I typically add a nudging to a monthly climatology in a band around the edge of the domain, ramping to zero in the middle. Note that this will overwrite TNUDG and friends in the default ROMS code.
Thanks Kate. I have read this discussion viewtopic.php?f=14&t=5200 and will try to see the new results.

I have another question. Is it appropriate to directly use soda as the initial condition for a long coupled simulation? Should I run ROMS separately for some decades of year to spin up?

Regards,
Yu

User avatar
kate
Posts: 4088
Joined: Wed Jul 02, 2003 5:29 pm
Location: CFOS/UAF, USA

Re: How to solve the speed trouble on the open boundaries?

#4 Unread post by kate »

I use SODA as my initial condition, no spin-up period. You can try a spin-up and see how much drift you get.

Post Reply