Number of history files greater than 9999

General scientific issues regarding ROMS

Moderators: arango, robertson

Post Reply
Message
Author
User avatar
gresendiz
Posts: 12
Joined: Mon Jun 18, 2018 7:01 pm
Location: CICESE

Number of history files greater than 9999

#1 Unread post by gresendiz »


Hello! I need some help; I hope someone can help me.
I have a simulation with output files greater than 9,999, but I don't know how to configure the ROMS to write the files without restart the simulation because the limit of history files is 9,999.

Thank you!
Gabriela :)

CICESE
PhD student

User avatar
kate
Posts: 4088
Joined: Wed Jul 02, 2003 5:29 pm
Location: CFOS/UAF, USA

Re: Number of history files greater than 9999

#2 Unread post by kate »

It's quite easy. Just change output.F so that you have:

Code: Select all

  10          FORMAT (a,'_',i5.5,'.nc')
instead of

Code: Select all

  10          FORMAT (a,'_',i4.4,'.nc')
and similar for all the other format statements.

User avatar
gresendiz
Posts: 12
Joined: Mon Jun 18, 2018 7:01 pm
Location: CICESE

Re: Number of history files greater than 9999

#3 Unread post by gresendiz »

Thank you for your help, Kate! I made the changes and will run the model. :D
Gabriela :)

CICESE
PhD student

User avatar
arango
Site Admin
Posts: 1347
Joined: Wed Feb 26, 2003 4:41 pm
Location: DMCS, Rutgers University
Contact:

Re: Number of history files greater than 9999

#4 Unread post by arango »

Yes, this is an easy change to make. However, making such a change will affect our nomenclature conventions for ensemble runs, 4D-Var forward solutions, operational set-ups, etc. I am reluctant to make such a change.

You are the first user since ROMS was released more than 20 years ago that creates over 10000 history files with the same base filename. That's a lot of I/O for a single solution and not that easy to manage. When we run applications for multiple years, we create a series of files where the year enters the filename to facilitate the management, analysis, and storage. ROMS has a perfect restart option, so the solution can be run for chunks of time. It is very unlikely that we create a 10000 counter in a single year. In the operational environment, we can have files like xxxx_roms_his_2022_0001.nc, and so on for solutions corresponding to year 2022. If we are doing forecasting, we usually have xxxx_roms_his_20220115.nc4 for a forecast issued on Jan 15, 2022. Here xxxx is a prefix to identify a particular application or region. As you can see, the user has full control over how the files are named and how many time records are included in each file. The I/O can be a bottleneck, so the user also takes into consideration how frequently to write fields into the average, diagnostics, floats, stations, history, quicksave, and restart files. When doing long solutions, we use the average file to record monthly, seasonal, and annual trends. The time-averaged file has important characteristics that you cannot see in the instantaneous fields.

User avatar
kate
Posts: 4088
Joined: Wed Jul 02, 2003 5:29 pm
Location: CFOS/UAF, USA

Re: Number of history files greater than 9999

#5 Unread post by kate »

He is at least the second person to run into this issue. I ran into it with daily output over 40 years, each day in its own file. Maybe it wouldn't happen with today's code, I don't know.

User avatar
arango
Site Admin
Posts: 1347
Joined: Wed Feb 26, 2003 4:41 pm
Location: DMCS, Rutgers University
Contact:

Re: Number of history files greater than 9999

#6 Unread post by arango »

Hi Kate, since ROMS now has a full-time management module (dateclock.F) that even includes a Fortran version of Matlab's datenum and datestr, it has been in my TODO list to add an option that includes some date format in the filename. Additionally, it also has day-of-the-year, ISO 8601 format, and others. I added all that capability when I coded the ESMF/NUOPC coupling interface. In my opinion, a date string is a way to go when having a large number of files in a solution. It is easy to identify a file for a particular date instead of trial and error by looking at the counter. The command ncdump -f -v ocean_time file.nc is also very useful.

User avatar
kate
Posts: 4088
Joined: Wed Jul 02, 2003 5:29 pm
Location: CFOS/UAF, USA

Re: Number of history files greater than 9999

#7 Unread post by kate »

Yes, someone in Enrique's lab wrote a Python code to rename files based on the date in there. Very useful!

MOM6 files all have names starting with the date, such as 20100101.ice_daily.nc

User avatar
gresendiz
Posts: 12
Joined: Mon Jun 18, 2018 7:01 pm
Location: CICESE

Re: Number of history files greater than 9999

#8 Unread post by gresendiz »

Thank you, Arango and Kate; I had a problem restarting my model in the past, so I tried running the model once time.
In this time, I was able to restart without a problem, and the two solutions helped me.



Thank you.
Gabriela
Gabriela :)

CICESE
PhD student

Post Reply