ROMS/TOMS - Output error ............ exit_flag: 3

General scientific issues regarding ROMS

Moderators: arango, robertson

Post Reply
Message
Author
ZhaoTH
Posts: 3
Joined: Mon Jan 18, 2021 5:14 pm
Location: Beijing Normal University

ROMS/TOMS - Output error ............ exit_flag: 3

#1 Unread post by ZhaoTH »

Hello ROMS experts,

I'm modeling the changes on Antarctica from 1996 to 2015 using WAOM10. But I met the following problem,

I used the forcing files of the whole period 1996-2015 to run the model, the running stopped at the first stpe with the error as shown as the log file(Attachemen file 1: ocean_waom10_log_1996_2015.txt).

Code: Select all

   STEP   Day HH:MM:SS  KINETIC_ENRG   POTEN_ENRG    TOTAL_ENRG    NET_VOLUME
          C => (i,j,k)       Cu            Cv            Cw         Max Speed

      0   152 02:00:00  7.058180E-04  2.027254E+04  2.027254E+04  6.408357E+16
          (173,296,19)  4.126555E-02  5.934571E-02  0.000000E+00  1.634354E+00
      DEF_HIS   - creating history file, Grid 01: /scratch/gh9/mx3818/waom/output_1996_2015/ocean_his_1996_2015_test1_0001.nc

 DEF_VAR - unable to define variable: Tobc_in
           in NetCDF file: /scratch/gh9/mx3818/waom/output_1996_2015/ocean_his_1996_2015_test1_0001.nc

 Elapsed CPU time (seconds):

 Node   # 47 CPU:      49.808
 Node   #  1 CPU:      51.386
 Node   #  2 CPU:      51.861
 Node   #  3 CPU:      50.420
 Node   # 33 CPU:      52.630
 Node   #  4 CPU:      51.964
 Node   #  6 CPU:      50.073
 Node   #  7 CPU:      48.978
 Node   #  8 CPU:      47.581
 Node   #  9 CPU:      49.910
 Node   # 10 CPU:      48.759
 Node   # 11 CPU:      49.182
 Node   # 12 CPU:      49.912
 Node   # 13 CPU:      49.711
 Node   # 14 CPU:      47.890
 Node   # 15 CPU:      47.818
 Node   # 16 CPU:      47.726
 Node   # 17 CPU:      49.544
 Node   # 18 CPU:      49.488
 Node   # 19 CPU:      48.337
 Node   # 20 CPU:      50.622
 Node   # 21 CPU:      48.010
 Node   # 22 CPU:      48.261
 Node   # 23 CPU:      47.516
 Node   # 24 CPU:      46.993
 Node   # 37 CPU:      50.395
 Node   # 25 CPU:      50.672
 Node   # 27 CPU:      50.133
 Node   # 28 CPU:      50.653
 Node   # 29 CPU:      50.364
 Node   # 30 CPU:      50.579
 Node   # 26 CPU:      50.405
 Node   # 38 CPU:      50.246
 Node   # 31 CPU:      50.047
 Node   # 34 CPU:      51.894
 Node   # 32 CPU:      49.794
 Node   #  5 CPU:      50.217
 Node   # 36 CPU:      51.928
 Node   # 35 CPU:      51.028
 Node   # 39 CPU:      50.364
 Node   # 40 CPU:      47.674
 Node   # 41 CPU:      49.918
 Node   # 42 CPU:      50.169
 Node   # 43 CPU:      49.846
 Node   # 44 CPU:      51.272
 Node   # 45 CPU:      50.018
 Node   # 46 CPU:      49.840
 Node   #  0 CPU:      37.019
 Node   #184 CPU:      47.974
 Node   #144 CPU:      51.584
 Node   #145 CPU:      55.849
 Node   #146 CPU:      56.330
 Node   #181 CPU:      50.827
 Node   #149 CPU:      54.669
 Node   #148 CPU:      56.098
 Node   #151 CPU:      53.066
 Node   #147 CPU:      54.594
 Node   #165 CPU:      51.952
 Node   #153 CPU:      53.125
 Node   #150 CPU:      54.709
 Node   #155 CPU:      52.436
 Node   #259 CPU:      56.792
 Node   #156 CPU:      53.097
 Node   #157 CPU:      52.236
 Node   #154 CPU:      52.946
 Node   #158 CPU:      51.918
 Node   #152 CPU:      51.552
 Node   #161 CPU:      53.661
 Node   #162 CPU:      54.079
 Node   #191 CPU:      49.885
 Node   #164 CPU:      53.002
 Node   #163 CPU:      52.222
 Node   # 99 CPU:      53.022
 Node   #167 CPU:      51.326
 Node   #142 CPU:      53.654
 Node   #166 CPU:      52.015
 Node   #143 CPU:      53.023
 Node   #169 CPU:      51.329
 Node   #122 CPU:      52.204
 Node   #170 CPU:      51.750
 Node   # 98 CPU:      54.881
 Node   #168 CPU:      49.120
 Node   #171 CPU:      50.937
 Node   #128 CPU:      54.281
 Node   #172 CPU:      51.658
 Node   #100 CPU:      55.540
 Node   #173 CPU:      50.911
 Node   #135 CPU:      54.140
 Node   #175 CPU:      50.425
 Node   #287 CPU:      52.994
 Node   #159 CPU:      52.346
 Node   #241 CPU:      52.734
 Node   #103 CPU:      51.612
 Node   #176 CPU:      49.400
 Node   # 94 CPU:      49.705
 Node   #242 CPU:      53.172
 Node   #104 CPU:      50.331
 Node   #177 CPU:      51.738
 Node   # 95 CPU:      49.316
 Node   #243 CPU:      52.623
 Node   # 97 CPU:      55.084
 Node   #227 CPU:      52.438
 Node   #174 CPU:      51.344
 Node   #244 CPU:      53.173
 Node   #105 CPU:      51.877
 Node   #231 CPU:      51.466
 Node   #179 CPU:      50.888
 Node   # 48 CPU:      49.862
 Node   #245 CPU:      52.624
 Node   #107 CPU:      51.173
 Node   #193 CPU:      54.088
 Node   #180 CPU:      51.603
 Node   # 49 CPU:      54.038
 Node   #246 CPU:      52.829
 Node   #108 CPU:      53.248
 Node   #217 CPU:      52.592
 Node   #182 CPU:      51.185
 Node   # 50 CPU:      53.374
 Node   #247 CPU:      51.853
 Node   #109 CPU:      51.587
 Node   #194 CPU:      53.182
 Node   #183 CPU:      50.384
 Node   # 51 CPU:      52.901
 Node   #261 CPU:      56.942
 Node   #110 CPU:      51.759
 Node   #224 CPU:      51.081
 Node   #178 CPU:      51.778
 Node   # 52 CPU:      53.914
 Node   #249 CPU:      52.050
 Node   #106 CPU:      52.180
 Node   #195 CPU:      52.319
 Node   #160 CPU:      51.231
 Node   # 54 CPU:      51.786
 Node   #248 CPU:      49.245
 Node   #111 CPU:      51.968
 Node   #198 CPU:      52.657
 Node   #185 CPU:      49.985
 Node   # 55 CPU:      51.790
 Node   #251 CPU:      51.687
 Node   #112 CPU:      49.876
 Node   #239 CPU:      50.913
 Node   #187 CPU:      49.868
 Node   # 88 CPU:      48.330
 Node   #252 CPU:      52.385
 Node   #113 CPU:      51.894
 Node   #200 CPU:      49.848
 Node   #188 CPU:      50.676
 Node   # 71 CPU:      50.848
 Node   #253 CPU:      51.512
 Node   #114 CPU:      51.978
 Node   #201 CPU:      51.840
 Node   #189 CPU:      50.132
 Node   # 57 CPU:      51.734
 Node   #262 CPU:      56.918
 Node   #115 CPU:      51.625
 Node   #202 CPU:      51.669
 Node   #190 CPU:      50.434
 Node   # 59 CPU:      51.504
 Node   #254 CPU:      51.821
 Node   #116 CPU:      52.635
 Node   #203 CPU:      51.474
 Node   #186 CPU:      50.999
 Node   # 60 CPU:      52.292
 Node   #286 CPU:      53.519
 Node   #117 CPU:      51.686
 Node   #204 CPU:      53.017
 Node   # 61 CPU:      51.433
 Node   #282 CPU:      54.001
 Node   #118 CPU:      51.483
 Node   #205 CPU:      51.320
 Node   # 62 CPU:      51.205
 Node   #258 CPU:      58.269
 Node   #101 CPU:      52.956
 Node   #192 CPU:      49.713
 Node   # 65 CPU:      53.054
 Node   #256 CPU:      54.049
 Node   #207 CPU:      51.817
 Node   # 70 CPU:      51.559
 Node   #260 CPU:      58.242
 Node   #238 CPU:      51.067
 Node   # 63 CPU:      51.097
 Node   #257 CPU:      58.169
 Node   #121 CPU:      52.171
 Node   #209 CPU:      52.982
 Node   # 90 CPU:      49.785
 Node   #263 CPU:      55.056
 Node   #120 CPU:      49.970
 Node   #206 CPU:      51.472
 Node   # 67 CPU:      52.296
 Node   #255 CPU:      51.376
 Node   #123 CPU:      51.603
 Node   #211 CPU:      51.981
 Node   # 58 CPU:      51.713
 Node   #264 CPU:      51.925
 Node   #124 CPU:      52.749
 Node   #212 CPU:      52.833
 Node   # 68 CPU:      53.488
 Node   #265 CPU:      54.524
 Node   #125 CPU:      51.612
 Node   #214 CPU:      51.866
 Node   # 66 CPU:      53.856
 Node   #266 CPU:      54.684
 Node   #102 CPU:      52.642
 Node   #210 CPU:      53.709
 Node   # 56 CPU:      49.469
 Node   #267 CPU:      53.904
 Node   #119 CPU:      50.626
 Node   #216 CPU:      49.586
 Node   # 53 CPU:      52.853
 Node   #268 CPU:      55.434
 Node   #126 CPU:      52.212
 Node   #229 CPU:      52.182
 Node   #270 CPU:      53.790
 Node   #129 CPU:      56.524
 Node   # 72 CPU:      47.750
 Node   #271 CPU:      53.083
 Node   #130 CPU:      56.620
 Node   #199 CPU:      51.166
 Node   # 73 CPU:      50.850
 Node   #269 CPU:      54.338
 Node   #131 CPU:      55.341
 Node   #219 CPU:      51.419
 Node   # 74 CPU:      51.100
 Node   #272 CPU:      53.995
 Node   #132 CPU:      56.871
 Node   #220 CPU:      52.802
 Node   # 75 CPU:      50.158
 Node   #273 CPU:      55.897
 Node   #127 CPU:      51.622
 Node   #221 CPU:      51.931
 Node   # 77 CPU:      50.368
 Node   #274 CPU:      56.141
 Node   #134 CPU:      54.825
 Node   #222 CPU:      51.915
 Node   # 78 CPU:      50.446
 Node   #276 CPU:      55.621
 Node   #136 CPU:      53.319
 Node   #225 CPU:      53.245
 Node   # 76 CPU:      51.414
 Node   #277 CPU:      54.483
 Node   #137 CPU:      53.888
 Node   #196 CPU:      53.828
 Node   # 80 CPU:      50.127
 Node   #275 CPU:      54.866
 Node   #138 CPU:      54.411
 Node   #223 CPU:      51.647
 Node   # 81 CPU:      51.655
 Node   #278 CPU:      54.545
 Node   #133 CPU:      54.473
 Node   #218 CPU:      52.501
 Node   # 79 CPU:      49.878
 Node   #279 CPU:      53.867
 Node   #139 CPU:      53.529
 Node   #226 CPU:      53.096
 Node   # 84 CPU:      53.300
 Node   #250 CPU:      51.991
 Node   #140 CPU:      54.593
 Node   #197 CPU:      52.652
 Node   # 85 CPU:      51.485
 Node   #281 CPU:      53.620
 Node   #141 CPU:      53.582
 Node   #228 CPU:      53.722
 Node   # 83 CPU:      51.332
 Node   #280 CPU:      51.422
 Node   # 96 CPU:      50.175
 Node   #230 CPU:      52.352
 Node   # 82 CPU:      52.492
 Node   #283 CPU:      53.218
 Node   #215 CPU:      50.555
 Node   # 87 CPU:      50.521
 Node   #284 CPU:      54.175
 Node   #213 CPU:      51.440
 Node   # 69 CPU:      52.827
 Node   #285 CPU:      53.521
 Node   #232 CPU:      49.159
 Node   # 89 CPU:      50.141
 Node   #240 CPU:      48.372
 Node   #233 CPU:      51.638
 Node   #234 CPU:      51.409
 Node   # 86 CPU:      51.824
 Node   #235 CPU:      51.084
 Node   # 91 CPU:      49.260
 Node   #236 CPU:      52.223
 Node   #237 CPU:      51.316
 Node   #208 CPU:      51.652
 Node   # 92 CPU:      50.369
 Node   # 93 CPU:      49.416
 Node   # 64 CPU:      51.448
 Total:             14943.855

 Nonlinear model elapsed time profile, Grid: 01

  Allocation and array initialization ..............        62.051  ( 0.4152 %)
  Ocean state initialization .......................         1.431  ( 0.0096 %)
  Reading of input data ............................      3812.617  (25.5129 %)
  Processing of input data .........................         1.326  ( 0.0089 %)
  Processing of output time averaged data ..........         0.278  ( 0.0019 %)
  Computation of vertical boundary conditions ......         0.140  ( 0.0009 %)
  Computation of global information integrals ......         0.812  ( 0.0054 %)
  Tidal forcing ....................................         1.746  ( 0.0117 %)
  2D/3D coupling, vertical metrics .................         1.337  ( 0.0089 %)
  Omega vertical velocity ..........................         0.501  ( 0.0034 %)
  Equation of state for seawater ...................         1.529  ( 0.0102 %)
  KPP vertical mixing parameterization .............         3.556  ( 0.0238 %)
                                              Total:      3887.324   26.0129

 Nonlinear model message Passage profile, Grid: 01

  Message Passage: 2D halo exchanges ...............         3.539  ( 0.0237 %)
  Message Passage: 3D halo exchanges ...............         2.220  ( 0.0149 %)
  Message Passage: 4D halo exchanges ...............         0.482  ( 0.0032 %)
  Message Passage: data broadcast ..................         0.350  ( 0.0023 %)
  Message Passage: data reduction ..................       320.743  ( 2.1463 %)
  Message Passage: boundary data gathering .........         1.088  ( 0.0073 %)
                                              Total:       328.423    2.1977

 All percentages are with respect to total time =        14943.855


 ROMS/TOMS - Output NetCDF summary for Grid 01:

 Analytical header files used:

     ROMS/Functionals/ana_btflux.h
     ROMS/Functionals/ana_srflux.h

 ROMS/TOMS - Output error ............ exit_flag:   3


 ERROR: Abnormal termination: NetCDF OUTPUT.
 REASON: NetCDF: Invalid dimension ID or name
I tried to reduce the simulation period. And I found that only I didn't load waom10_nudge_2012(2013,2014,2015).nc, the running can continue. So I set the simulation period as 1996_2011,and rerun the model. It worked and the running stopped at the step of the year 2011,with output netcdf files and the log file (Attachemen file 2: ocean_waom10_log_1996_2011.txt).

But once the forcing files contains the nudge_2012.nc file, the running will stop in the first step with similar log file as attachment 1.

However, my colleagues and I checked all of my nudge files and look at the fields using ncview and attributes/variables names using ncdump -h command, we can’t see anything wrong with them. I even copied nudge_ 2011.nc as nudge_ 2012.nc (only changed the ocean time), but the result had no changes.

I have no other way to try, so I come here for help. Could you please give me some information on this? Any idea will be much appreciated! Thanks in advance!

Best Regards,
Meng
Attachments
Attachment 2. ocean_waom10_log_1996_2011.txt
(91.18 MiB) Downloaded 236 times
Attachment 1. ocean_waom10_log_1996_2015.txt
(139.96 KiB) Downloaded 151 times

User avatar
kate
Posts: 4088
Joined: Wed Jul 02, 2003 5:29 pm
Location: CFOS/UAF, USA

Re: ROMS/TOMS - Output error ............ exit_flag: 3

#2 Unread post by kate »

Your error might not be in the input files at all. It says:
DEF_VAR - unable to define variable: Tobc_in
in NetCDF file: /scratch/gh9/mx3818/waom/output_1996_2015/
Do you have permission to write to this /scratch/gh9/mx3818/waom/ directory? Does it exist?

ZhaoTH
Posts: 3
Joined: Mon Jan 18, 2021 5:14 pm
Location: Beijing Normal University

Re: ROMS/TOMS - Output error ............ exit_flag: 3

#3 Unread post by ZhaoTH »

Hey,

Thanks for your reply.

Yes, this folder /scratch/gh9/mx3818/waom/ is where my waom model is located. I'm sure I have write permission. In addition, the output file path in my Attachment 2 is/scratch/gh9/mx3818/waom/output_1996_2011/,which worked.

Code: Select all

Output/Input Files:

             Output Restart File:  /scratch/gh9/mx3818/waom/output_1996_2011/ocean_rst_1996_2011_test1.nc
        Prefix for History Files:  /scratch/gh9/mx3818/waom/output_1996_2011/ocean_his_1996_2011_test1
       Prefix for Averages Files:  /scratch/gh9/mx3818/waom/output_1996_2011/ocean_avg_1996_2011_test1

In fact, I also encountered similar errors in my another tests (same forcing files):

Please See Attachment 3. test1_log.txt

Code: Select all

DEF_VAR - unable to define variable: x_psi
           in NetCDF file: ocean_his_1996_2011_0001.nc
Please See Attachment 4. test2_log.txt

Code: Select all

 DEF_VAR - unable to define variable: h
           in NetCDF file: /scratch/gh9/mx3818/waom/output_1996_2012/ocean_his_1996_2012_test1_0001.nc
Many thanks!

Best,
Meng
Attachments
Attachment 4. test2_log.txt
(138.35 KiB) Downloaded 158 times
Attachment 3. test1_log.txt
(138.11 KiB) Downloaded 165 times

User avatar
wilkin
Posts: 875
Joined: Mon Apr 28, 2003 5:44 pm
Location: Rutgers University
Contact:

Re: ROMS/TOMS - Output error ............ exit_flag: 3

#4 Unread post by wilkin »

I see you are using #define PARALLEL_IO

That is almost certainly the issue. Try running without that and see if the error occurs.
John Wilkin: DMCS Rutgers University
71 Dudley Rd, New Brunswick, NJ 08901-8521, USA. ph: 609-630-0559 jwilkin@rutgers.edu

Post Reply