I spin up the North Pacfic experiment with open boundary and surface force interpolated from ECCO2 data. The model have been run for 1 year, and the output frequency is 1 month. During the 1-year run, the model result is fine. Then,I want to check the oceanic state at 19th in the February. However, when I restarted the model with the history file at 12:00 1st February, the model blowed up after 5 steps (the original model have been run for 1 year, and never blow up). I also tried to run the model by extract the oceanic state from history file to an initial file, but the model also blowed up.
So I want to know, why the original run never blowed up? However, when I restarted the model from history or restart file, the model blowed up in 5 steps.
Thanks for any suggestion.
Blow up when restart from history file
Re: Blow up when restart from history file
I have run from an averages file, no problem. Do you have all the fields you need? Did you tell ROMS it was a new run with NRREC=0? If there's biology, did you really save everything you need in the history file? I've had runs die with COBALT when the averages file didn't have say the htotal field which it then assumed was all zero, leading to big trouble. You need to figure out what's wrong with your restart runs.
Re: Blow up when restart from history file
Hi Kate,kate wrote: ↑Thu Sep 09, 2021 6:50 pm I have run from an averages file, no problem. Do you have all the fields you need? Did you tell ROMS it was a new run with NRREC=0? If there's biology, did you really save everything you need in the history file? I've had runs die with COBALT when the averages file didn't have say the htotal field which it then assumed was all zero, leading to big trouble. You need to figure out what's wrong with your restart runs.
Thank you very much for your suggestion.
There maybe some misunderstanding in my previous statement. I restarted the model from the historical file, not from the average file. When I extract the historical data from the history file to the initial file, I actually run with NRREC==0. There are just dynamical cpps without any biology in my run. In the new run, the variables in the initial file include zeta, u, v, temp , salt, ubar, vbar. I think I have all the fields I need. Should I add other variables in the initial file? I found that I have some ANA cpps, such as ANA_SPONGE, is it possible that these ANA apps account for the blow-up when restarting the model with history file or remaking initial file?
My cppdef is shown as following:
#define SOLVE3D
#define UV_COR
#define UV_ADV
#define UV_U3HADVECTION
#undef UV_C4VADVECTION
#undef TS_U3HADVECTION
#define TS_C4VADVECTION
#define Q_CORRECTION
#define PERFECT_RESTART
#define UV_VIS2
#define TS_DIF2
#define SALINITY
#define MIX_S_UV /* use if mixing along constant S-surfaces */
#define NONLIN_EOS
#define MIX_S_TS#define DJ_GRADPS /* pressure gradient term using splines density Jacobian (Shchepetkin,2000) */
#define UV_QDRAG
#define VISC_GRID
#define DIFF_GRID
#define QCORRECTION /* use if net heat flux correction */
#define SRELAXATION /* use if freshwater flux correction */
#define CURVGRID
#define MASKING
#define SPHERICAL
#define AVERAGES
#define SPLINES
#define SOLAR_SOURCE
#define MY25_MIXING
#ifdef MY25_MIXING
# undef N2S2_HORAVG /* use if horizontal smoothing of buoyancy/shear */
# define KANTHA_CLAYSON /* use if kantha and clayson stability function */
#endif
#define ANA_SPONGE
#define RADIATION_2D
#define SSH_TIDES
#ifdef SSH_TIDES
# define ADD_FSOBC
#endif
#define UV_TIDES
#ifdef UV_TIDES
# define ADD_M2OBC
#endif
#define RAMP_TIDES
#define ANA_BSFLUX
#define ANA_BTFLUX
Re: Blow up when restart from history file
Hi Kate,kate wrote: ↑Thu Sep 09, 2021 6:50 pm I have run from an averages file, no problem. Do you have all the fields you need? Did you tell ROMS it was a new run with NRREC=0? If there's biology, did you really save everything you need in the history file? I've had runs die with COBALT when the averages file didn't have say the htotal field which it then assumed was all zero, leading to big trouble. You need to figure out what's wrong with your restart runs.
I found another interesting thing. Now I have shorten the time step DT from 300s to 150s and run the model for one year again. After that, I restart my model from day 33 12:00:00 with history file. Now, the restarting model don't blow up anymore. However, the output of the restarting run is quite different from that of the original run. I found that in the restarting run, the model do not read w-velocity from history file (actually,the w-velocity has been written in the history file). We can see that, in the original run the w-velocity of step 17280 at point (0328,345,11) is 3.588361E-01, and in the restarting run the w-velocity is 0.000000E+00. After the restarting model begin to run, the output is also different. I can't understand why, shouldn't the output of original and restarting run be the same?
original run
17280 33 12:00:00 1.830313E-02 2.202263E+04 2.202265E+04 3.150442E+17
(0328,345,11) 1.768172E-02 2.471083E-02 3.588361E-01 3.303913E+00
17281 33 12:02:30 1.830477E-02 2.202264E+04 2.202266E+04 3.150443E+17
(0328,345,11) 1.770010E-02 2.470716E-02 3.590542E-01 3.303713E+00
17282 33 12:05:00 1.830637E-02 2.202265E+04 2.202267E+04 3.150444E+17
(0328,345,11) 1.771756E-02 2.470290E-02 3.592529E-01 3.303500E+00
17283 33 12:07:30 1.830793E-02 2.202266E+04 2.202268E+04 3.150445E+17
(0328,345,11) 1.773411E-02 2.469807E-02 3.594314E-01 3.303274E+00
17284 33 12:10:00 1.830945E-02 2.202267E+04 2.202269E+04 3.150446E+17
(0328,345,11) 1.774981E-02 2.469275E-02 3.595905E-01 3.303035E+00
17285 33 12:12:30 1.831093E-02 2.202268E+04 2.202269E+04 3.150446E+17
(0328,345,11) 1.776473E-02 2.468698E-02 3.597321E-01 3.302783E+00
restarting run
17280 33 12:00:00 1.830311E-02 2.202263E+04 2.202265E+04 3.150442E+17
(0388,367,37) 3.009061E-02 3.151716E-02 0.000000E+00 3.303912E+00
17281 33 12:02:30 1.830496E-02 2.202264E+04 2.202266E+04 3.150443E+17
(0328,345,11) 1.769985E-02 2.470711E-02 3.584562E-01 3.303719E+00
17282 33 12:05:00 1.830681E-02 2.202265E+04 2.202267E+04 3.150444E+17
(0328,345,11) 1.779485E-02 2.479339E-02 3.592483E-01 3.303895E+00
17283 33 12:07:30 1.830865E-02 2.202266E+04 2.202268E+04 3.150445E+17
(0328,345,11) 1.788681E-02 2.488050E-02 3.611593E-01 3.304087E+00
17284 33 12:10:00 1.831045E-02 2.202267E+04 2.202269E+04 3.150446E+17
(0328,345,11) 1.797985E-02 2.496994E-02 3.625994E-01 3.304280E+00
17285 33 12:12:30 1.831220E-02 2.202268E+04 2.202269E+04 3.150446E+17
(0328,345,11) 1.807359E-02 2.506148E-02 3.638682E-01 3.304478E+00
Re: Blow up when restart from history file
I'm glad you got it to run. As for the restarts matching exactly, you do not have enough information stored in the history files to accomplish that. The nature of the ROMS timestepping requires information from more than one timelevel of some fields, plus some right-hand-side information to do perfect restarts. That's why there's a PERFECT_RESTART option which may or may not give you perfect restarts, but your chances without that are nil.
As for PERFECT_RESTART, it ought to work. One way to break it is to take an odd number of steps before writing the restart file. ROMS does not save the value of nrhs and friends, merely assuming that you will take an even number of steps between restarts, in which case it should work. If it still isn't a perfect restart, there are ways to debug it - I went through it for the seaice component which had been imperfect.
As for PERFECT_RESTART, it ought to work. One way to break it is to take an odd number of steps before writing the restart file. ROMS does not save the value of nrhs and friends, merely assuming that you will take an even number of steps between restarts, in which case it should work. If it still isn't a perfect restart, there are ways to debug it - I went through it for the seaice component which had been imperfect.
Re: Blow up when restart from history file
Hi Kate,kate wrote: ↑Fri Sep 10, 2021 5:43 pm I'm glad you got it to run. As for the restarts matching exactly, you do not have enough information stored in the history files to accomplish that. The nature of the ROMS timestepping requires information from more than one timelevel of some fields, plus some right-hand-side information to do perfect restarts. That's why there's a PERFECT_RESTART option which may or may not give you perfect restarts, but your chances without that are nil.
As for PERFECT_RESTART, it ought to work. One way to break it is to take an odd number of steps before writing the restart file. ROMS does not save the value of nrhs and friends, merely assuming that you will take an even number of steps between restarts, in which case it should work. If it still isn't a perfect restart, there are ways to debug it - I went through it for the seaice component which had been imperfect.
Thanks for your detailed explanation, I got a thorough understanding.
Thank you again for your helpful suggestion.