Blow up in Second step
-
- Posts: 27
- Joined: Sat Apr 07, 2018 12:59 am
- Location: INDIAN INSTITUTE OF TECHNOLOGY KHARAGPUR
Blow up in Second step
Dear All
I'm stuck at one point where I'm trying to run a standalone simulation for one month using ROMS_3.7. I have tried two different grids and it is showing diag speed trouble for one grid and a blow up in the first step for the other grid. My DT is 30 seconds. I'm trying to initialise the model using INCOIS GODAS data for the month of May 2020. I could successfully prepare the initial and boundary files using matlab tool box. However for two different types of grid, I'm getting a blow up and diag speed error. I guess there is some fundamental error. I'm attaching the log files of runs using both grids. Kindly help me with this. Previously I have done simulations with the same model, have not encountered these kind of errors.
I'm stuck at one point where I'm trying to run a standalone simulation for one month using ROMS_3.7. I have tried two different grids and it is showing diag speed trouble for one grid and a blow up in the first step for the other grid. My DT is 30 seconds. I'm trying to initialise the model using INCOIS GODAS data for the month of May 2020. I could successfully prepare the initial and boundary files using matlab tool box. However for two different types of grid, I'm getting a blow up and diag speed error. I guess there is some fundamental error. I'm attaching the log files of runs using both grids. Kindly help me with this. Previously I have done simulations with the same model, have not encountered these kind of errors.
- Attachments
-
roms_test1.txt
- (78.98 KiB) Downloaded 210 times
-
roms_test2.txt
- (75.59 KiB) Downloaded 212 times
Re: Blow up in Second step
I see a number of things in your logfiles that are a bit unconventional and you might consider adjusting.
Both runs have both these defined:
I recommend you start with only #undef UV_VIS4 until to show you need it. Especially since you have #define UV_DRAG_GRID and #define ANA_DRAG_GRID. Do you really have an analytical specification of spatially varying linear drag? If that's on purpose, check that those coefficients are sensible and you don't have a parallel bug in the analytical code crossing tile boundaries.
You still have #define UPWELLING which activates certain options in the ana_*.h functionals and made be overshadowing your own customizations (like the bottom drag). It is customary to set one's own ROMS_APPLICATION name and use that in build_roms.sh (export ROMS_APPLICATION=UPWELLING) in place of the examples. Look at Build/analytical.f90 to check if UPWELLING is giving you code you did not intend.
You have Flather and Chapman OBC on ubar and zeta, and analytical boundary ubar,vbar (in roms_test2.txt) but clamped u,v. Those can't possibly be consistent (i.e vertical average of u,v won't be consistent with ubar,vbar. I recommend radiation on u,v until you get this running. Or, vertically average your u,v in ROMS discrete coordinates and make sure that is the ubar,vbar in the boundary file.
You have clamped T,S boundary conditions, yet the range of values on e.g. the east boundary is
That zero might be in the land mask, but be careful that you aren't clamping T to zero at one water point and a value closer to 32 immediately next to it. That will introduce very strong horizontal density gradients along the boundary.
You mix advection schemes on the active tracers.
That's not fundamentally wrong, but we have seen this to give unstable behavior. Try making these the same until you get this running.
In test 1 you have both of
Just use #define MIX_GEO_TS. You don't want both (ROMS might not give you both, but don't chance it).
Is there a good reason to use this?
Start simple and #undef that.
Be careful for #define SRELAXATION. There are some posts of mine from last year with a user having trouble with this option. Again, start without so many options activated until you get a starting stable run.
The fact that you have a NaN in the kinetic energy on the first time step ...
says to me you initialized a NaN in velocity, possibly in the land mask but that is still not allowed. If you have a _FillValue in your initial conditions velocity that may unwittingly introduce a NaN in velocity.
Both runs have both these defined:
Code: Select all
UV_VIS2 Harmonic mixing of momentum
UV_VIS4 Biharmonic mixing of momentum
You still have #define UPWELLING which activates certain options in the ana_*.h functionals and made be overshadowing your own customizations (like the bottom drag). It is customary to set one's own ROMS_APPLICATION name and use that in build_roms.sh (export ROMS_APPLICATION=UPWELLING) in place of the examples. Look at Build/analytical.f90 to check if UPWELLING is giving you code you did not intend.
You have Flather and Chapman OBC on ubar and zeta, and analytical boundary ubar,vbar (in roms_test2.txt) but clamped u,v. Those can't possibly be consistent (i.e vertical average of u,v won't be consistent with ubar,vbar. I recommend radiation on u,v until you get this running. Or, vertically average your u,v in ROMS discrete coordinates and make sure that is the ubar,vbar in the boundary file.
You have clamped T,S boundary conditions, yet the range of values on e.g. the east boundary is
Code: Select all
GET_NGFLD_NF90 - temperature eastern boundary condition, 2020-05-01 00:00:00.00
(Grid= 01, Rec=1, Index=2, File: roms_bry_amphan.nc)
(Tmin= 7426.0000 Tmax= 7456.0000) t = 7426.0000
(Min = 0.00000000E+00 Max = 3.20222588E+01)
You mix advection schemes on the active tracers.
Code: Select all
Tracer Advection Scheme: NLM
========================
Variable Grid Horizontal Vertical
--------- ---- ------------ ------------
temp 1 Upstream3 Centered4
salt 1 HSIMT HSIMT
In test 1 you have both of
Code: Select all
MIX_GEO_TS Mixing of tracers along geopotential surfaces
MIX_S_TS Mixing of tracers along constant S-surfaces
Is there a good reason to use this?
Code: Select all
VISC_3DCOEF Horizontal, time-dependent 3D viscosity coefficient
Be careful for #define SRELAXATION. There are some posts of mine from last year with a user having trouble with this option. Again, start without so many options activated until you get a starting stable run.
The fact that you have a NaN in the kinetic energy on the first time step ...
Code: Select all
TIME-STEP YYYY-MM-DD hh:mm:ss.ss KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME
C => (i,j,k) Cu Cv Cw Max Speed
0 2020-05-01 00:00:00.00 NaN 1.630508E+04 NaN 9.985486E+15
(063,163,40) 1.679527E-03 2.074747E-03 0.000000E+00 8.896352E-01
John Wilkin: DMCS Rutgers University
71 Dudley Rd, New Brunswick, NJ 08901-8521, USA. ph: 609-630-0559 jwilkin@rutgers.edu
71 Dudley Rd, New Brunswick, NJ 08901-8521, USA. ph: 609-630-0559 jwilkin@rutgers.edu
-
- Posts: 27
- Joined: Sat Apr 07, 2018 12:59 am
- Location: INDIAN INSTITUTE OF TECHNOLOGY KHARAGPUR
Re: Blow up in Second step
Thank you very much for your very detailed reply. I will try these and will post the outcome of my run in the same section. Good day!
-
- Posts: 27
- Joined: Sat Apr 07, 2018 12:59 am
- Location: INDIAN INSTITUTE OF TECHNOLOGY KHARAGPUR
Re: Blow up in Second step
Dear Sir
I couldn't understand what this means. However, I have changed the conditions to RadNud for T,S, U and V. I have made changes to the run mentioned in test2 (as per your suggestions) and this is the log file. As per your suggestion, I have renamed my application file to BOB to avoid mismatch in the options. The point (035,17,32) in which the speed is beyond 20 m/s looks like water point only. Am i missing some fundamental point?You have Flather and Chapman OBC on ubar and zeta, and analytical boundary ubar,vbar (in roms_test2.txt) but clamped u,v. Those can't possibly be consistent (i.e vertical average of u,v won't be consistent with ubar,vbar. I recommend radiation on u,v until you get this running. Or, vertically average your u,v in ROMS discrete coordinates and make sure that is the ubar,vbar in the boundary file.
You have clamped T,S boundary conditions, yet the range of values on e.g. the east boundary is
- Attachments
-
roms_bob1.txt
- (67.87 KiB) Downloaded 203 times
Re: Blow up in Second step
Now you have deactivated all horizontal mixing. You have neither #define TS_DIF2 not #define TS_DIF4, which is why no horizontal mixing coefficient values nl_tnu2 are reported to the log. #define MIX_GEO_TS
You do have some horizontal viscosity (#define MIX_GEO_UV) but the viscosity coefficient is zero in the log:
I recommend #define MIX_S_UV (not GEO). You had that before but changed it.
You choice of momentum advection scheme is notoriously noisy
and with zero viscosity to control the noisiness I would expect this to blow-up, which it did. Just use the default advection scheme for momentum.
In a change from the log for test2, this latest is reporting old cpp defs that have been removed
which explains why the log no longer reports the advection scheme. Looks like you compiled a different version of the code for this experiment.
In test2 you had STATIONS and now you have no stations but have FLOATS. Wait until you get your run stable before you mess with all that. I'm using diff to compare your log files and it's tricky to make sense of it all with all these changes.
Yes, you should look at what's going on at point (35,17,32). You'll want to plot at every time step because vertical velocity is going nuts right away.
But add some horizontal viscosity first. That should help.
You do have some horizontal viscosity (#define MIX_GEO_UV) but the viscosity coefficient is zero in the log:
Code: Select all
0.0000E+00 nl_visc2 NLM Horizontal, harmonic mixing coefficient
(m2/s) for momentum.
You choice of momentum advection scheme is notoriously noisy
Code: Select all
UV_C2ADVECTION Second-order centered differences advection of momentum
In a change from the log for test2, this latest is reporting old cpp defs that have been removed
Code: Select all
TS_C4HADVECTION Fourth-order centered horizontal advection of tracers
TS_C4VADVECTION Fourth-order centered vertical advection of tracers
In test2 you had STATIONS and now you have no stations but have FLOATS. Wait until you get your run stable before you mess with all that. I'm using diff to compare your log files and it's tricky to make sense of it all with all these changes.
Yes, you should look at what's going on at point (35,17,32). You'll want to plot at every time step because vertical velocity is going nuts right away.
You might to adjust the bathymetry or land mask there. It's impossible to tell from the logs.TIME-STEP YYYY-MM-DD hh:mm:ss.ss KINETIC_ENRG POTEN_ENRG TOTAL_ENRG NET_VOLUME
C => (i,j,k) Cu Cv Cw Max Speed
0 2020-05-13 00:00:00.00 9.447909E-04 1.493786E+04 1.493786E+04 6.898608E+15
(192,006,32) 1.724965E-03 2.002071E-03 0.000000E+00 9.554885E-01
1 2020-05-13 00:00:30.00 1.171937E-03 1.511131E+04 1.511131E+04 6.819455E+15
(166,047,32) 7.491161E-03 8.240924E-03 7.429613E-06 3.397028E+00
2 2020-05-13 00:01:00.00 1.749324E-03 1.511126E+04 1.511126E+04 6.819455E+15
(035,017,32) 1.676149E-04 5.337860E-04 1.095465E+03 6.325016E+00
3 2020-05-13 00:01:30.00 2.615535E-03 1.511118E+04 1.511118E+04 6.819454E+15
But add some horizontal viscosity first. That should help.
John Wilkin: DMCS Rutgers University
71 Dudley Rd, New Brunswick, NJ 08901-8521, USA. ph: 609-630-0559 jwilkin@rutgers.edu
71 Dudley Rd, New Brunswick, NJ 08901-8521, USA. ph: 609-630-0559 jwilkin@rutgers.edu
-
- Posts: 27
- Joined: Sat Apr 07, 2018 12:59 am
- Location: INDIAN INSTITUTE OF TECHNOLOGY KHARAGPUR
Re: Blow up in Second step
I tried with all of these. now the MODEL runs for 200 time steps and then stops. Could someone tell what could be the possible reasons for the increase in maximum speed. My initial and boundary files seems to be fine and without NaN's. I'm doing the simulation for BoB region with south, west open boundary and closed north and east boundaries. The forcing are also fine and are provided every six hour. Have tried running with dt =30 and dt=10 I would like to know what else should be taken care while setting up the simulations. I'm attaching the log file.
- Attachments
-
roms_new.txt
- (93.47 KiB) Downloaded 196 times
Re: Blow up in Second step
The log gives you some guidance that the model is blowing up near point i=225, j=61, k=23.196 2020-05-01 00:32:40.00 3.869347E-01 1.555930E+04 1.555969E+04 8.325429E+15
(225,061,23) 3.047532E-04 2.696655E-04 5.290505E-01 1.944507E+01
197 2020-05-01 00:32:50.00 3.923971E-01 1.555928E+04 1.555967E+04 8.325429E+15
(225,061,23) 3.276896E-04 2.693862E-04 5.415630E-01 1.996086E+01
198 2020-05-01 00:33:00.00 3.979175E-01 1.555925E+04 1.555965E+04 8.325429E+15
(225,061,23) 3.519085E-04 2.693302E-04 5.541660E-01 2.049019E+01
Found Error: 01 Line: 298 Source: ROMS/Nonlinear/main3d.F
Found Error: 01 Line: 298 Source: ROMS/Drivers/nl_ocean.h
Blowing-up: Saving latest model state into RESTART file
WRT_RST - wrote re-start fields (Index=1,1) in record = 0000003
ROMS writes the model state at blow-up to record 3 of your RESTART file.
So, plot those data and see what is going on. Usually these issues are solved by modifying the bathymetry or land/sea mask near points that are causing extreme velocity. It does not look like you have strong horizontal velocities, so it's probably vertical. Some smoothing of steep bathymetry might be in order.
John Wilkin: DMCS Rutgers University
71 Dudley Rd, New Brunswick, NJ 08901-8521, USA. ph: 609-630-0559 jwilkin@rutgers.edu
71 Dudley Rd, New Brunswick, NJ 08901-8521, USA. ph: 609-630-0559 jwilkin@rutgers.edu
-
- Posts: 27
- Joined: Sat Apr 07, 2018 12:59 am
- Location: INDIAN INSTITUTE OF TECHNOLOGY KHARAGPUR
Re: Blow up in Second step
Dear All
I'm attaching the screen shot of my grids made using 25 km resolution for whole NIO with east and north closed and for Bay of Bengal with 9 km resolution keeping east and north closed. I have used the modified ETOPO which most of literature suggests. I have used theta_s and theta_b as 7 and 0.1 with N=40 and TCline as 250. I have even tried with TCline = 150. For both these grids keeping the options similar, I have used dt = 10 ad dt = 30. While the run progresses for abput 300 time steps with dt = 10 , it stops after 100 time steps with dt = 30. I have been trying different grid making softwares such as roms_tools and gridbuilder for exactly solving the issue. I suspect that it could be the problem of the grid while the boundary and initial files looks ok. The forcing values are also reasonable. The only issue could be grid or the bathymetry. As suggested , I have used modified ETOPO instead of standard ETOPO which has adapted by most of the studies in these regions. Kindly help and I would be forever grateful if someone could help me solve this issue.
I'm attaching the screen shot of my grids made using 25 km resolution for whole NIO with east and north closed and for Bay of Bengal with 9 km resolution keeping east and north closed. I have used the modified ETOPO which most of literature suggests. I have used theta_s and theta_b as 7 and 0.1 with N=40 and TCline as 250. I have even tried with TCline = 150. For both these grids keeping the options similar, I have used dt = 10 ad dt = 30. While the run progresses for abput 300 time steps with dt = 10 , it stops after 100 time steps with dt = 30. I have been trying different grid making softwares such as roms_tools and gridbuilder for exactly solving the issue. I suspect that it could be the problem of the grid while the boundary and initial files looks ok. The forcing values are also reasonable. The only issue could be grid or the bathymetry. As suggested , I have used modified ETOPO instead of standard ETOPO which has adapted by most of the studies in these regions. Kindly help and I would be forever grateful if someone could help me solve this issue.
- Attachments
-
- ncview.mask_u_nio.ps
- (442.87 KiB) Downloaded 188 times
-
- ncview_h_bob.ps ncview_bob_h.ps
- (522.23 KiB) Downloaded 198 times
-
- ncview.mask_u_nio.ps
- (442.87 KiB) Downloaded 174 times
-
- ncview.mask_u_bob.ps
- (520.25 KiB) Downloaded 207 times
-
- Posts: 27
- Joined: Sat Apr 07, 2018 12:59 am
- Location: INDIAN INSTITUTE OF TECHNOLOGY KHARAGPUR
Re: Blow up in Second step
Dear Dr Wilkin,
I have solved this error. The issue was with the making of initial file and I have rectified it. Now the model is running fine. Me and my team would like to thank you for your valuable suggestions which has helped us in improving our set up.
I have solved this error. The issue was with the making of initial file and I have rectified it. Now the model is running fine. Me and my team would like to thank you for your valuable suggestions which has helped us in improving our set up.
Re: Blow up in Second step
Dear Sun_92,
I'm a new user of ROMS and got a very similar issue with you when I plan to try different horizontal resolutions. My model setup works fine for 1 and 2km horizontal resolution, but blows up in the second time step in coarser resolution (5 and 10km). My DT is 50s. So could u plz have look about the log file of 5km version?
Thank you so much
![Laughing :lol:](./images/smilies/icon_lol.gif)
Best regards
JZ
- Attachments
-
fort.37.txt
- (480.13 KiB) Downloaded 217 times
-
- Posts: 27
- Joined: Sat Apr 07, 2018 12:59 am
- Location: INDIAN INSTITUTE OF TECHNOLOGY KHARAGPUR
Re: Blow up in Second step
Hello
The options used and the values of input files are not seen in the log. Without that it is not possible for me to figure out what exactly goes wrong and why the blow up. Usually if the speed is above 20 m/s, there will be a blow up which can be adjusted by smoothing the bathymetry and by checking for bad values in your input files. Schemes used may also give you spurious results, therefore try to change the options as mentioned in the post by Dr Wilkin and make sure all your input files are good and without bad values. Also please plot and see the point which gives you the blow up to see if something is going wrong.
The options used and the values of input files are not seen in the log. Without that it is not possible for me to figure out what exactly goes wrong and why the blow up. Usually if the speed is above 20 m/s, there will be a blow up which can be adjusted by smoothing the bathymetry and by checking for bad values in your input files. Schemes used may also give you spurious results, therefore try to change the options as mentioned in the post by Dr Wilkin and make sure all your input files are good and without bad values. Also please plot and see the point which gives you the blow up to see if something is going wrong.
Re: Blow up in Second step
Sorry Im a new user of ROMS, maybe my .in file covers what you want?
Yes, Im following Dr Wilkin's suggestion and trying to find where is the bad value. By a quick check, max u and v are not exceed 7m/s.
Thanks
Yes, Im following Dr Wilkin's suggestion and trying to find where is the bad value. By a quick check, max u and v are not exceed 7m/s.
Thanks
- Attachments
-
ocean_isomip_plus_ocn3.in
- (65.26 KiB) Downloaded 215 times
Re: Blow up in Second step
Also, what I want to is to test model sensitivity to different resolution. So I keep everything the same, run the model with 1km, 2km, 5km and 10km resolution. It works fine for 1 and 2km but blows up for 5 and 10km in ~58S. I changed the TNU2 and VISC2 but didnt help.