How to download HYCOM GLBa0.08 expt60.5 data?

[s193264]

Dear ROMS Users

I have a question about creating climatology data.
I would like to download the 2005 data in HYCOM data GLBa expt60.5 using COAWST/Tools/mfiles/roms_clm/roms_master_climatology_coawst_mw.m in a MATLAB 2022a environment.
So, I edited and executed the url, wdr, and modelgrid parts of the .m file.
(EX. url ='http://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_60.5/')

However, the following error occurs.

[>> roms_master_climatology_coawst_mw


getting roms grid, hycom grid, and overlapping indices
getting roms grid dimensions ...
getting HYCOM grid data from http://tds.hycom.org/thredds/dodsC/GLBa ... 60.5/2005/

Error using matlab.internal.imagesci.netcdflib
The NetCDF library encountered an error during execution of 'open' function - 'Access failure (-77)'.

Error in netcdf.open (line 77)
[varargout{:}] = matlab.internal.imagesci.netcdflib('open', ...

Error in internal.matlab.imagesci.nc/openToRead (line 1290)
this.ncRootid = netcdf.open(this.Filename,'NOWRITE');

Error in internal.matlab.imagesci.nc (line 124)
this.openToRead();

Error in ncread (line 69)
ncObj = internal.matlab.imagesci.nc(ncFile);

Error in get_ijrg (line 45)
hycom_lon=ncread(url,'lon');

Error in roms_master_climatology_coawst_mw (line 59)
[gn, clm]=get_ijrg(url, modelgrid, theta_s, theta_b, Tcline, N, Vtransform, Vstretching);]


Has anyone encountered the same problem?
I would be grateful if you could let me know if there are any mistakes.
thank you.

[jcwarner]

the tools we have there are for aggregated data

https://tds.hycom.org/thredds/catalog.html
i dont see the 60.5 on that list.

-j

[s193264]

Thank you for reply

[s193264]

the tools we have there are for aggregated data

https://tds.hycom.org/thredds/catalog.html
i dont see the 60.5 on that list.

-j

Dear warner

Certainly, there doesn't seem to be 60.5 in that list.
How should I get the HYCOM data needed to create the 2005 clm data?
Should I use GLBu0.08 expt_19.1?
If so, what should I change other than "url" in roms_master_climatology_coawst_mw.m?
I changed it to "url = https: //tds.hycom.org/thredds/dodsC/GLBu0.08/expt_19.1/2005" and tried to execute it, but I still get an error.
If you can tell me, I will be very helpful.

Thank you.

[s193264]

the tools we have there are for aggregated data

https://tds.hycom.org/thredds/catalog.html
i dont see the 60.5 on that list.

-j

Dear warner

Certainly, there doesn't seem to be 60.5 in that list.
How should I get the HYCOM data needed to create the 2005 clm data?
Should I use GLBu0.08 expt_19.1?
If so, what should I change other than "url" in roms_master_climatology_coawst_mw.m?
I changed it to "url = https: //tds.hycom.org/thredds/dodsC/GLBu0.08/expt_19.1/2005" and tried to execute it, but I still get an error.
If you can tell me, I will be very helpful.

Thank you.

Sorry,
The previous URL was incorrect, and when I added / 2005 / to the end, the program was executed and I was able to access the HYCOM data.
However, I am in trouble because a different error occurs.

[getting roms grid, hycom grid, and overlapping indices
getting roms grid dimensions ...
getting HYCOM grid data from http://tds.hycom.org/thredds/dodsC/GLBu ... _19.1/2005
getting roms grid dimensions ...
optimizing grid dimensions ...
going to create clm file
getting the number of time records ...
creating netcdf file coawst_clm.nc
## Defining Global Attributes...
## Defining Dimensions...
## Defining Variables, and Attributes...
Interpolating u for 27-Aug-2005
doing griddata u for HYCOM level 1
Warning: Duplicate data points have been detected and removed - corresponding values have been averaged.
> In updatclim_coawst_mw (line 67)
In roms_master_climatology_coawst_mw (line 64)
doing griddata u for HYCOM level 2
catch u Unable to download HYCOM u data at17-Jul-2022 23:08:30
doing griddata u for HYCOM level 1]

[jcwarner]

doing griddata u for HYCOM level 1
...
doing griddata u for HYCOM level 2
catch u Unable to download HYCOM u data at17-Jul-2022 23:08:30

if it worked for level 1, but not for level 2, that is an internet or Thredds issue. this is common.
you can try again, or try some other way.
Also, if you want to use ECMWF data, you can follow John Wilkin's methods here:
viewtopic.php?p=23429&hilit=era5#p23429

[jcwarner]

oops, not enough coffee yet. those ecmwf is for bulk fluxes.

there are other posts on getting hycom data, such as:
viewtopic.php?p=23944&hilit=hycom#p23944
viewtopic.php?p=23938&hilit=hycom#p23938
viewtopic.php?p=23929&hilit=hycom#p23929

[s193264]

oops, not enough coffee yet. those ecmwf is for bulk fluxes.

there are other posts on getting hycom data, such as:
viewtopic.php?p=23944&hilit=hycom#p23944
viewtopic.php?p=23938&hilit=hycom#p23938
viewtopic.php?p=23929&hilit=hycom#p23929

Dear warner

Thank you for your kind advice.
But I started the thread. So I have been facing this HYCOM undownloadable issue for a long time.
Therefore, I tried seapy, but an error occurred, so I am in the process of consulting with the developer Brian.

I have a question,
Can HYCOM be downloaded only from the aggregate catalog(http://tds.hycom.org/thredds/catalog/da ... talog.html) ?
What I mean is is it possible to use "roms_master_climatology_coawst_mw.m" to download the GLBa/expt_60.5 data in "All data" of "Unaggregated" in HYCOM? (http://tds.hycom.org/thredds/catalog/da ... talog.html)

[arief121]

Dear ROMS friends,

I want to ask similar question about the HYCOM Data.
Actually I used the same roms_master_climatology_coawst_mw.m to generate the climatology data for ROMS through MATLAB. I tried 2 deafferents url,

1) first, I used http://tds.hycom.org/thredds/dodsC/GLBu0.08/expt_93.0 with the script
T1 = datenum(2018,9,20,12,0,0); %start date
numdays = 10;
dayFrequency = 1;
url = 'http://tds.hycom.org/thredds/dodsC/GLBu0.08/expt_93.0';

and when I run it, I encounter the errors like this:
%==================================================
getting roms grid, hycom grid, and overlapping indices
getting roms grid dimensions ...
getting HYCOM grid data from http://tds.hycom.org/thredds/dodsC/GLBu0.08/expt_93.0
Error using internal.matlab.imagesci.nc/getGroupAndVarid (line 2076)
Could not find variable or group 'X' in file.

Error in internal.matlab.imagesci.nc/read (line 594)
[gid, varid] = getGroupAndVarid(this, location);

Error in ncread (line 59)
vardata = ncObj.read(varName, varargin{:});

Error in get_ijrg (line 38)
numX=ncread(url,'X');

Error in roms_master_climatology_coawst_mw (line 66)
[gn, clm]=get_ijrg(url, modelgrid, theta_s, theta_b, Tcline, N, Vtransform, Vstretching);
%==================================================

2) second, I used http://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.9 with the script

T1 = datenum(2012,10,28,12,0,0); %start date
numdays = 10;
dayFrequency = 1;
url = 'http://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.9';

and when I run it, I encounter other errors like this:

%==================================================
getting roms grid, hycom grid, and overlapping indices
getting roms grid dimensions ...
getting HYCOM grid data from http://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.9
getting roms grid dimensions ...
optimizing grid dimensions ...
going to create clm file
getting the number of time records ...
creating netcdf file coawst_clm.nc
## Defining Global Attributes...
## Defining Dimensions...
## Defining Variables, and Attributes...
Interpolating u for 28-Oct-2012
doing griddata u for HYCOM level 1
catch u Unable to download HYCOM u data at27-Oct-2022 11:11:10
doing griddata u for HYCOM level 1
catch u Unable to download HYCOM u data at27-Oct-2022 11:11:22
doing griddata u for HYCOM level 1
catch u Unable to download HYCOM u data at27-Oct-2022 11:11:36
.
.
.
catch u Unable to download HYCOM u data at27-Oct-2022 11:31:14
doing griddata u for HYCOM level 1
catch u Unable to download HYCOM u data at27-Oct-2022 11:31:16
Caught "std::exception" Exception message is:
Message Catalog MATLAB:FileIO was not loaded from the file. Please check file location, format or contents

%==================================================

I wait for several times but the message always appear and it stop with no output data generated.

Has this issue ever happened to anyone else? Please let me know if there are any errors;
I'd appreciate it.
Many thanks.

[jcwarner]

so i just tried the second one you have, and it worked for me:

disp('getting roms grid, hycom grid, and overlapping indices')
[gn, clm]=get_ijrg(url, modelgrid, theta_s, theta_b, Tcline, N, Vtransform, Vstretching);
getting roms grid, hycom grid, and overlapping indices
getting roms grid dimensions ...
getting HYCOM grid data from http://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.9
getting roms grid dimensions ...
optimizing grid dimensions ...
>> disp('going to create clm file')
fn=updatclim_coawst_mw(T1, gn, clm, 'coawst_clm.nc', wdr, url)
going to create clm file
getting the number of time records ...
creating netcdf file coawst_clm.nc
## Defining Global Attributes...
## Defining Dimensions...
## Defining Variables, and Attributes...
Interpolating u for 28-Oct-2012
doing griddata u for HYCOM level 1
doing griddata u for HYCOM level 2
..
doing griddata u for HYCOM level 26
...
doing griddata v for HYCOM level 2
doing griddata salt for HYCOM level 33
....
Finished creating clim file at 31-Oct-2022 09:27:22
fn = 'coawst_clm.nc'


Some time HYCOM Thredds takes a while.
you can always go here:
http://tds.hycom.org/thredds/catalog.html

and look for the data set you need.

[remit]

so i just tried the second one you have, and it worked for me:

disp('getting roms grid, hycom grid, and overlapping indices')
[gn, clm]=get_ijrg(url, modelgrid, theta_s, theta_b, Tcline, N, Vtransform, Vstretching);
getting roms grid, hycom grid, and overlapping indices
getting roms grid dimensions ...
getting HYCOM grid data from http://tds.hycom.org/thredds/dodsC/GLBa0.08/expt_90.9
getting roms grid dimensions ...
optimizing grid dimensions ...
>> disp('going to create clm file')
fn=updatclim_coawst_mw(T1, gn, clm, 'coawst_clm.nc', wdr, url)
going to create clm file
getting the number of time records ...
creating netcdf file coawst_clm.nc
## Defining Global Attributes...
## Defining Dimensions...
## Defining Variables, and Attributes...
Interpolating u for 28-Oct-2012
doing griddata u for HYCOM level 1
doing griddata u for HYCOM level 2
..
doing griddata u for HYCOM level 26
...
doing griddata v for HYCOM level 2
doing griddata salt for HYCOM level 33
....
Finished creating clim file at 31-Oct-2022 09:27:22
fn = 'coawst_clm.nc'


Some time HYCOM Thredds takes a while.
you can always go here:
http://tds.hycom.org/thredds/catalog.html

and look for the data set you need.

Dear, Warner
Your solution works for the second question, but for the new type HYCOM data set (GLBy0.08/expt_93.0) the script (roms_master_climatology_coawst_mw.m) it's not working. Because the script needs the variable X, Y, and MT which is not available in the new HYCOM data set. I need a solution to this. Tks

[jcwarner]

i updated the script (get_ijrg.m) a few months ago (?) to use:

%
% Read HYCOM lon lat depth
%
display(['getting HYCOM grid data from ', url])
lonlat_fullgrid=1;
try
numX=ncread(url,'X');
numY=ncread(url,'Y');
hycom_lon=ncread(url,'Longitude');
hycom_lat=ncread(url,'Latitude');
hycom_depth=ncread(url,'Depth');
catch
hycom_lon=ncread(url,'lon');
hycom_lat=ncread(url,'lat');
hycom_depth=ncread(url,'depth');
lonlat_fullgrid=0;
numX=length(hycom_lon);
numY=length(hycom_lat);
end

So it should look for the older style first, if there is an X Y, and if it fails, then it goes to get the newer style of lon lat.

can you give that a try?

[mariefrahman]

Hi all,

anybody knows why this error occur? I tried to set up the HYCOM as boundary condition using this script : roms_master_climatology_coawst_mw.m. My domain for this simulation is in Java Sea (Indoenesia region). however when i run the matlab script, it run but repeated warnings appear and data cannot be processed like this:

getting roms grid, hycom grid, and overlapping indices
10:16:55getting roms grid dimensions ...
getting HYCOM grid data from https://tds.hycom.org/thredds/dodsC/GLBy0.08/expt_93.0
getting roms grid dimensions ...
optimizing grid dimensions ...
going to create clm file
10:17:07getting the number of time records ...
creating netcdf file coawst_clm.nc
## Defining Global Attributes...
## Defining Dimensions...
## Defining Variables, and Attributes...
Interpolating u for 01-May-2023
doing griddata u for HYCOM level 1
doing griddata u for HYCOM level 2
doing griddata u for HYCOM level 3
doing griddata u for HYCOM level 4
doing griddata u for HYCOM level 5
doing griddata u for HYCOM level 6
doing griddata u for HYCOM level 7
doing griddata u for HYCOM level 8
doing griddata u for HYCOM level 9
doing griddata u for HYCOM level 10
doing griddata u for HYCOM level 11
doing griddata u for HYCOM level 12
doing griddata u for HYCOM level 13
doing griddata u for HYCOM level 14
doing griddata u for HYCOM level 15
doing griddata u for HYCOM level 16
doing griddata u for HYCOM level 17
doing griddata u for HYCOM level 18
doing griddata u for HYCOM level 19
doing griddata u for HYCOM level 20
Warning: The underlying triangulation is empty - the points may be collinear.
> In griddata>useScatteredInterp (line 183)
In griddata (line 120)
In maplev (line 28)
In updatclim_coawst_mw (line 74)
In roms_master_climatology_coawst_mw_karimun9km (line 69)
catch u Unable to download HYCOM u data at24-Aug-2023 10:17:28
doing griddata u for HYCOM level 1
doing griddata u for HYCOM level 2
doing griddata u for HYCOM level 3
doing griddata u for HYCOM level 4
doing griddata u for HYCOM level 5
doing griddata u for HYCOM level 6
doing griddata u for HYCOM level 7
doing griddata u for HYCOM level 8
doing griddata u for HYCOM level 9
doing griddata u for HYCOM level 10
doing griddata u for HYCOM level 11
doing griddata u for HYCOM level 12
doing griddata u for HYCOM level 13
doing griddata u for HYCOM level 14
doing griddata u for HYCOM level 15
doing griddata u for HYCOM level 16
doing griddata u for HYCOM level 17
doing griddata u for HYCOM level 18
doing griddata u for HYCOM level 19
doing griddata u for HYCOM level 20
Warning: The underlying triangulation is empty - the points may be collinear.
> In griddata>useScatteredInterp (line 183)
In griddata (line 120)
In maplev (line 28)
In updatclim_coawst_mw (line 74)
In roms_master_climatology_coawst_mw_karimun9km (line 69)
catch u Unable to download HYCOM u data at24-Aug-2023 10:17:38

this is the input from roms_master_climatology script I used in this case:

%%%%%%%%%%%%%%%%%%%%% START OF USER INPUT %%%%%%%%%%%%%%%%%%%%%%%%%%

% (1) Enter start date (T1) and number of days to get climatology data
T1 = datenum(2023,5,1,00,0,0); %start date
%number of days and frequency to create climatology files for
numdays = 60;
dayFrequency = 3;

% (2) Enter URL of the HYCOM catalog for the requested time, T1
% see http://tds.hycom.org/thredds/catalog.html

url = 'https://tds.hycom.org/thredds/dodsC/GLBy0.08/expt_93.0'; % Hindcast Data: Dec-04-2018 to Present *3-hourly*

% (3) Enter working directory (wdr)
wdr = 'C:\Arief\ROMS\05_ICBC\04_karimun_jawa';

% (4) Enter path and name of the ROMS grid
modelgrid = 'grid_karimun_9km_v4.nc';

% (5) Enter grid vertical coordinate parameters --These need to be consistent with the ROMS setup.
theta_s = 5.0;
theta_b = 0.4;
Tcline = 20.0;
N = 20;
Vtransform = 2;
Vstretching = 4;

%%%%%%%%%%%%%%%%%%%%% END OF USER INPUT %%%%%%%%%%%%%%%%%%%%%%%%%%

hereby I attach my roms_grid file.

Thank you for the help.
Arief

[jcwarner]

a couple of things here.

1 - there is a depth in your grid exactly = 0 (h(52,4) ==0) and that caused some troubles in the routine to comute ubar and vbar

cc=roms_zint_mw(u,gn); ubar=rho2u_2d_mw(u2rho_2d_mw(cc)./gn.h);
cc=roms_zint_mw(v,gn); vbar=rho2v_2d_mw(v2rho_2d_mw(cc)./gn.h);
%== Rotate the velocity
uv=(u2rho_2d_mw(ubar)+sqrt(-1)*v2rho_2d_mw(vbar)).*theta;
ubar=rho2u_2d_mw(real(uv)); vbar=rho2v_2d_mw(imag(uv));

the ./gn.h created an Inf. so change that grid bathy location to be 0.5 or something.

2- there was an issue in maplev. The area you are modeling is rather shallow, and at N=20 the data from Hycom was sparse. the maplev function returned a empty matrix as it tried to fill the array. I made a change to maplev and pushed out a new version.

[mariefrahman]

Hi J,

thank you for your correction about the grid file. I change it into new grid and make sure there are no more zero or minus values in the grid file. and also I tried you new maplev script but the error still the same. Do you know why?

Warning: The underlying triangulation is empty - the points may be collinear.
> In griddata>useScatteredInterp (line 183)
In griddata (line 120)
In maplev (line 28)

I sent the new grid file I used.