Having trouble getting lagrangian float working...

Discussion of how to use ROMS on different regional and basin scale applications.

Moderators: arango, robertson

Post Reply
Message
Author
cjroach
Posts: 14
Joined: Thu Jul 19, 2018 3:22 pm
Location: MetOcean Solutions

Having trouble getting lagrangian float working...

#1 Unread post by cjroach »

Okay, firs time posting here. I'm a post-doc with MetOcean Solutions in NZ involved in the Kiwis' MOANA project which looks at biological connectivity of a number of key commercial marine species. I'm specifically working on how climate change is going to change ocean circulation which first involves using ROMS to downscale several medium (2030-2050ish) and long (2080-2100) term CMIP6 scenarios then conducting Lagrangian analysis using the resulting velocity fields. Problem is I've got lotsa experiance on the Lagrangian analysis side but this is my first time doing "serious" numerical modelling, so I'm rather learning on the fly.

As a prelude to running the future climate downscalings (current plan is mid-term and long-term downscaling for a high emissions pathway and a long-term downscaling for a medium emissions pathway) I'm running some test cases using the current MOANA "backbone" historical configuration. One of these involves releasing particles in ROMS to get a "full" temporal resolution "ground truth" before conducting the same releases in an offline Lagrangian tracking package (probably OpenDrift, maybe PARCELS) with coarser temporal resolution in order to scope out the acceptable output time interval that captures all the key dynamics while not producing too excessively big history files.

The problem is, when I add floats to the (setting MY_CPP_FLAGS="${MY_CPP_FLAGS} -DFLOATS" and adding # define FLOATS to the relevant .h file per the test example included in ROMS) ROMS runs but the resulting *flt*.nc file contains NaNs in all lat, lon, depth etc. variables. The first obvious possibility is I've released floats into the terrain (as the first test per if float are working I'm using a single release location), but I've very specifically chosen a location well offshore and confined my release to the upper 20 m. The second possibility is I've misunderstood something about how release locations and times are specified, and got something wrong in the floats.in file (included in the code block below). The third option is there's something out of whack with my configuration and need to modify the .h file further (that said I'd have expected a crash instead of just NaN values in outputs...).

Note that I'm using ROMS/TOMS v3.9 (SVN Rev 1049) on the NZ NESI HPC system (Linux, x86_64) with a 395x465x50 grid split over 100 nodes.

Code: Select all

! Switch to control the computation of floats trajectories within nested
! and/or multiple connected grids, [1:Ngrids].
!
     Lfloats == T

! Switch to control the printing of floats positions to standard output
! file, [1:Ngrids].
!
      Fprint == T

! Flag indicating re-start from previous solution. If FRREC = 0, a new
! NetCDF output file is created.

       FRREC == 0

! Number of floats to release in each nested grid.  These values are
! essential because the FLOATS structure in "mod_floats" is dynamically
! allocated using these values, [1:Ngrids].

     NFLOATS == 240

! Initial floats locations for all grids:
!
!   G      Nested grid number
!   C      Initial horizontal coordinate type (0: grid units, 1: spherical)
!   T      Float trajectory type (1: Lagrangian, 2: isobaric, 3: Geopotential)
!   N      Number floats to be released at (Fx0,Fy0,Fz0)
!   Ft0    Float release time (days) after model initialization
!   Fx0    Initial float X-location (grid units or longitude)
!   Fy0    Initial float Y-location (grid units or latitude)
!   Fz0    Initial float Z-location (grid units or depth)
!   Fdt    Float cluster release time interval (days)
!   Fdx    Float cluster X-distribution parameter
!   Fdy    Float cluster Y-distribution parameter
!   Fdz    Float cluster Z-distribution parameter

POS = G, C, T, N,   Ft0,    Fx0,    Fy0,    Fz0,    Fdt,    Fdx,    Fdy,   Fdz

1 1 1 240   0  175.0  -45.0 -20.00 0 0 0 0

User avatar
wilkin
Posts: 875
Joined: Mon Apr 28, 2003 5:44 pm
Location: Rutgers University
Contact:

Re: Having trouble getting lagrangian float working...

#2 Unread post by wilkin »

It's a bit of a wild guess, but possibly having Fcount > 0 but all of Fdt=Fdx=Fdy=Fdz=0 is messing things up.

It looks like you are trying to release all 240 floats at the same location and time. They will all go the same place unless you also #define VWALK_FORWARD.

Perhaps try spreading them out using non-zero values of Fdt, Fdx, Fdy, Fdz to see if some of them start correctly.

It is also worth experimenting with Fz0 of different sign, so that you specify start points in both z (as you have, with Fz0 = -20.0) but also in s-index, such as with Fz0 = 1 and Fdz = 1 (so you start a float at every vertical level). Do this with several input lines each having the appropriate Fcount.

Is there a THREDDS server with model output in ROMS native file format? The Backbone version 2 output at the site I have bookmarked has all the grid coordinates stripped off. Private Message me if you want me to do a couple of tests for you using full output files.

BTW: If you have #define FLOATS in the .h file you don't need MY_CPP_FLAGS="${MY_CPP_FLAGS} -DFLOATS".
John Wilkin: DMCS Rutgers University
71 Dudley Rd, New Brunswick, NJ 08901-8521, USA. ph: 609-630-0559 jwilkin@rutgers.edu

cjroach
Posts: 14
Joined: Thu Jul 19, 2018 3:22 pm
Location: MetOcean Solutions

Re: Having trouble getting lagrangian float working...

#3 Unread post by cjroach »

Thanks John, I'll try your suggestions.

Per MOANA output, as far as I'm aware "full" versions of the latest variant of the backbone configuration aren't public yet. At the moment I think everything is stored on NESI, and thus only really accessible to people involved in the project.
MetOcean intends to set up an NZ based THREDDS server to host the MOANA model output eventually, but that process has only really just started... and sending through a raw *hist*.nc file isn't too practical (~250GB per in-model month...).

If your suggestions don't work I'll have a chat with Joao about if tests using full output are necessary.

cjroach
Posts: 14
Joined: Thu Jul 19, 2018 3:22 pm
Location: MetOcean Solutions

Re: Having trouble getting lagrangian float working...

#4 Unread post by cjroach »

Well, none of those options seem to have helped.

Looking through a log file containing all the text output it does look like ROMS is both initializing and running the particle tracking (taking up about 5% of CPU time per below).

Code: Select all

 Nonlinear model elapsed CPU time profile, Grid: 01

  Allocation and array initialization ..............        11.945  ( 0.0016 %)
  Ocean state initialization .......................        43.279  ( 0.0057 %)
  Reading of input data ............................      6484.270  ( 0.8604 %)
  Processing of input data .........................     18898.235  ( 2.5076 %)
  Processing of output time averaged data ..........      4978.815  ( 0.6606 %)
  Computation of vertical boundary conditions ......       312.408  ( 0.0415 %)
  Computation of global information integrals ......     12771.026  ( 1.6945 %)
  Writing of output data ...........................    186742.260  (24.7783 %)
  Model 2D kernel ..................................    117755.934  (15.6247 %)
  Lagrangian floats trajectories ...................     35171.373  ( 4.6668 %)
  Tidal forcing ....................................      5831.488  ( 0.7738 %)
  2D/3D coupling, vertical metrics .................     17997.029  ( 2.3880 %)
  Omega vertical velocity ..........................      9254.003  ( 1.2279 %)
  Equation of state for seawater ...................     22168.225  ( 2.9414 %)
  Atmosphere-Ocean bulk flux parameterization ......      7953.139  ( 1.0553 %)
  GLS vertical mixing parameterization .............    145128.408  (19.2567 %)
  3D equations right-side terms ....................     24871.789  ( 3.3002 %)
  3D equations predictor step ......................     54575.475  ( 7.2415 %)
  Pressure gradient ................................      8567.778  ( 1.1368 %)
  Corrector time-step for 3D momentum ..............     29701.879  ( 3.9411 %)
  Corrector time-step for tracers ..................     36789.819  ( 4.8815 %)
                                              Total:    746008.578   98.9857 %

  Unique kernel(s) regions profiled ................    746008.578   98.9857 %
  Residual, non-profiled code ......................      7644.628    1.0143 %
Output float netcdf file contains variables lon and lat, implying that (as is correct in this case) we're using a spherical grid. Was wondering if there might be something messing up in def_floats.h, but can't see any obvious reason why that'd happen.

Though, on reading notes n the code, I stumbled across it' handling or restart files. And since we're using a restart file as the start point for our tests... That could be the problem.

Updated FRREC in the float input file to -1 and will test ASAP.

Edit: That didn't help.

One thing I have noticed is all examples in the/Test/ folder that employ grid coordinates in meters instead of lat-lon...

User avatar
m.hadfield
Posts: 521
Joined: Tue Jul 01, 2003 4:12 am
Location: NIWA

Re: Having trouble getting lagrangian float working...

#5 Unread post by m.hadfield »

> One thing I have noticed is all examples in the/Test/ folder that employ grid coordinates in meters instead of lat-lon...

Hello CJ. Greetings from my home in sunny Welllington.

I suggest you try specifying float locations in grid coordinates and see if that works.

The other thing to look at is the start time: are your floats being released? The float release time Ft0 either does or doesn't take account of DSTART: I forget which, but I recall it being a point of confusion for me.

Having Fdt=Fdx=Fdy=Fdz=0 should not cause your problem, though all your floats will have the same trajectory.

cjroach
Posts: 14
Joined: Thu Jul 19, 2018 3:22 pm
Location: MetOcean Solutions

Re: Having trouble getting lagrangian float working...

#6 Unread post by cjroach »

Hi m.hadfield,
Yes, I have tried with grid-coordinates instead of lat-lon.

DSTART? That could be the issue, I'll double-check what we've got and see if adding it to Ft0 fixes things.

cjroach
Posts: 14
Joined: Thu Jul 19, 2018 3:22 pm
Location: MetOcean Solutions

Re: Having trouble getting lagrangian float working...

#7 Unread post by cjroach »

Hi again Mark,
Tried a few test just with the flt_test case changing DSTART and TIME_REF, neither of which seem to affect float start times.

User avatar
m.hadfield
Posts: 521
Joined: Tue Jul 01, 2003 4:12 am
Location: NIWA

Re: Having trouble getting lagrangian float working...

#8 Unread post by m.hadfield »

I suspect it's all a matter of time. ROMS has a variable called time (usually appearing in the code as time(ng)) representing time in seconds relative to a reference. (The variable TIME_REF can then be used to specify the calendar date-time of this reference.) It also has a variable tdays(ng) equal to the same time, but in days. Many model runs, such as the float test example, start with time = 0, but many do not. If you are initialising your model run from a restart file, then it will probably not be starting at time = 0; it will instead be starting at the time associated with the restart record.

It never hurts to check these things out in the code. If you look at ROMS/Nonlinear/step_floats.F at around line 702, you will see code to release each float if the time is within half a time step of Tinfo(itstr,l), where l is the float counter. And if you look at at ROMS/Utility/read_fltpar.F you will see code to set Tinfo(itstr,:) based on the values of Ft0 and Fdt read in from the floats input file. So, yes, what I've said above seems to be true.

So, look at the ROMS output for the model run you are experimenting with and establish what range of times it covers. Then choose a suitable values of Ft0.

cjroach
Posts: 14
Joined: Thu Jul 19, 2018 3:22 pm
Location: MetOcean Solutions

Re: Having trouble getting lagrangian float working...

#9 Unread post by cjroach »

Thanks Mark,
That seems to have got float running.

-Chris

Post Reply