PYROMS: SCRIP installation error

[Phadtaya]

Dear all,

I recently start learning ROMS and try to use pyroms to plot my results. I installed pyroms following the direction on https://github.com/ESMG/pyroms and got stuck (i.e. errors) during the scrip package installation. This is what I got:

Code: Select all

$ export PREFIX=/Users/ppoem/miniconda3/envs/py3/lib/python3.7/site-packages/netcdf-shared
$ make DEVELOP=1 PREFIX=$PREFIX install

f2py --fcompiler=gnu95 --f90exec=gfortran --f90flags='-g -fdefault-real-8 -ffixed-form -O2 -fPIC' \
	-L. -L/Users/ppoem/miniconda3/envs/py3/lib/python3.7/site-packages/netcdf-shared/lib -lnetcdf -lnetcdff -ljpeg -lmfhdf -ldf -lhdf5 -lhdf5_fortran -lhdf5_hl -lhdf5hl_fortran -I. -I/Users/ppoem/miniconda3/envs/py3/lib/python3.7/site-packages/netcdf-shared/include -m scrip -c kinds_mod.o constants.o iounits.o netcdf.o grids.o remap_vars.o remap_distwgt.o remap_conserv.o remap_bilinear.o remap_bicubic.o remap_read.o remap.o timers.o remap_write.o  ./pyscrip.f90
Reading .f2py_f2cmap ...
	Mapping "integer(kind=int_kind)" to "int"
	Mapping "real(kind=dbl_kind)" to "double"
	Mapping "logical(kind=log_kind)" to "char"
Successfully applied user defined changes from .f2py_f2cmap
running build
running config_cc
unifing config_cc, config, build_clib, build_ext, build commands --compiler options
running config_fc
unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options
running build_src
build_src
building extension "scrip" sources
f2py options: []
f2py:> /var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/src.macosx-10.9-x86_64-3.7/scripmodule.c
creating /var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/src.macosx-10.9-x86_64-3.7
Reading fortran codes...
	Reading file './pyscrip.f90' (format:fix)
Post-processing...
	Block: scrip
			Block: compute_remap_weights
In: :scrip:./pyscrip.f90:compute_remap_weights
get_useparameters: no module kinds_mod info used by compute_remap_weights
In: :scrip:./pyscrip.f90:compute_remap_weights
get_useparameters: no module constants info used by compute_remap_weights
In: :scrip:./pyscrip.f90:compute_remap_weights
get_useparameters: no module iounits info used by compute_remap_weights
In: :scrip:./pyscrip.f90:compute_remap_weights
get_useparameters: no module timers info used by compute_remap_weights
In: :scrip:./pyscrip.f90:compute_remap_weights
get_useparameters: no module grids info used by compute_remap_weights
In: :scrip:./pyscrip.f90:compute_remap_weights
get_useparameters: no module remap_vars info used by compute_remap_weights
In: :scrip:./pyscrip.f90:compute_remap_weights
get_useparameters: no module remap_conservative info used by compute_remap_weights
In: :scrip:./pyscrip.f90:compute_remap_weights
get_useparameters: no module remap_distance_weight info used by compute_remap_weights
In: :scrip:./pyscrip.f90:compute_remap_weights
get_useparameters: no module remap_bilinear info used by compute_remap_weights
In: :scrip:./pyscrip.f90:compute_remap_weights
get_useparameters: no module remap_bicubic info used by compute_remap_weights
In: :scrip:./pyscrip.f90:compute_remap_weights
get_useparameters: no module remap_write info used by compute_remap_weights
			Block: test_remap_weights
In: :scrip:./pyscrip.f90:test_remap_weights
get_useparameters: no module kinds_mod info used by test_remap_weights
In: :scrip:./pyscrip.f90:test_remap_weights
get_useparameters: no module constants info used by test_remap_weights
In: :scrip:./pyscrip.f90:test_remap_weights
get_useparameters: no module iounits info used by test_remap_weights
In: :scrip:./pyscrip.f90:test_remap_weights
get_useparameters: no module netcdf_mod info used by test_remap_weights
In: :scrip:./pyscrip.f90:test_remap_weights
get_useparameters: no module grids info used by test_remap_weights
In: :scrip:./pyscrip.f90:test_remap_weights
get_useparameters: no module remap_vars info used by test_remap_weights
In: :scrip:./pyscrip.f90:test_remap_weights
get_useparameters: no module remap_mod info used by test_remap_weights
In: :scrip:./pyscrip.f90:test_remap_weights
get_useparameters: no module remap_read info used by test_remap_weights
			Block: remap
In: :scrip:./pyscrip.f90:remap
get_useparameters: no module kinds_mod info used by remap
In: :scrip:./pyscrip.f90:remap
get_useparameters: no module constants info used by remap
Post-processing (stage 2)...
Building modules...
	Building module "scrip"...
		Constructing wrapper function "compute_remap_weights"...
		  compute_remap_weights(namelist_file)
		Constructing wrapper function "test_remap_weights"...
		  test_remap_weights(namelist_file)
		Constructing wrapper function "remap"...
		  remap(dst_array,map_wts,dst_add,src_add,src_array,[num_links,src_grid_size,dst_grid_size,src_grad1,src_grad2,src_grad3])
	Wrote C/API module "scrip" to file "/var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/src.macosx-10.9-x86_64-3.7/scripmodule.c"
  adding '/var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/src.macosx-10.9-x86_64-3.7/fortranobject.c' to sources.
  adding '/var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/src.macosx-10.9-x86_64-3.7' to include_dirs.
copying /Users/ppoem/miniconda3/envs/py3/lib/python3.7/site-packages/numpy/f2py/src/fortranobject.c -> /var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/src.macosx-10.9-x86_64-3.7
copying /Users/ppoem/miniconda3/envs/py3/lib/python3.7/site-packages/numpy/f2py/src/fortranobject.h -> /var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/src.macosx-10.9-x86_64-3.7
build_src: building npy-pkg config files
running build_ext
customize UnixCCompiler
customize UnixCCompiler using build_ext
customize Gnu95FCompiler
Found executable /usr/local/bin/gfortran
customize Gnu95FCompiler using build_ext
building 'scrip' extension
compiling C sources
C compiler: gcc -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/ppoem/miniconda3/envs/py3/include -arch x86_64 -I/Users/ppoem/miniconda3/envs/py3/include -arch x86_64

creating /var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/var
creating /var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/var/folders
creating /var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/var/folders/7_
creating /var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn
creating /var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T
creating /var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy
creating /var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/src.macosx-10.9-x86_64-3.7
compile options: '-I. -I/Users/ppoem/miniconda3/envs/py3/lib/python3.7/site-packages/netcdf-shared/include -I/var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/src.macosx-10.9-x86_64-3.7 -I/Users/ppoem/miniconda3/envs/py3/lib/python3.7/site-packages/numpy/core/include -I/Users/ppoem/miniconda3/envs/py3/include/python3.7m -c'
gcc: /var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/src.macosx-10.9-x86_64-3.7/scripmodule.c
gcc: /var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/src.macosx-10.9-x86_64-3.7/fortranobject.c
In file included from In file included from /var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/src.macosx-10.9-x86_64-3.7/scripmodule.c/var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/src.macosx-10.9-x86_64-3.7/fortranobject.c::162:
:
In file included from In file included from /var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/src.macosx-10.9-x86_64-3.7/fortranobject.h/var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/src.macosx-10.9-x86_64-3.7/fortranobject.h::1313:
:
In file included from In file included from /Users/ppoem/miniconda3/envs/py3/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h/Users/ppoem/miniconda3/envs/py3/lib/python3.7/site-packages/numpy/core/include/numpy/arrayobject.h::44:
:
In file included from In file included from /Users/ppoem/miniconda3/envs/py3/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h/Users/ppoem/miniconda3/envs/py3/lib/python3.7/site-packages/numpy/core/include/numpy/ndarrayobject.h::1212:
:
In file included from In file included from /Users/ppoem/miniconda3/envs/py3/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h/Users/ppoem/miniconda3/envs/py3/lib/python3.7/site-packages/numpy/core/include/numpy/ndarraytypes.h::18301830:
:
/Users/ppoem/miniconda3/envs/py3/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h/Users/ppoem/miniconda3/envs/py3/lib/python3.7/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h::1717::22::  warningwarning: : "Using"Using

            deprecateddeprecated  NumPyNumPy  API,API,  disabledisable  itit  withwith  "          "#define"          "#define  NPY_NO_DEPRECATED_APINPY_NO_DEPRECATED_API  NPY_1_7_API_VERSION"NPY_1_7_API_VERSION"  [-W#warnings][-W#warnings]

#warning "Using deprecated NumPy API, disable it with " \#warning "Using deprecated NumPy API, disable it with " \

 ^ ^

/var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/src.macosx-10.9-x86_64-3.7/scripmodule.c:142:12: warning: unused function
      'f2py_size' [-Wunused-function]
static int f2py_size(PyArrayObject* var, ...)
           ^
2 warnings generated.
1 warning generated.
compiling Fortran sources
Fortran f77 compiler: /usr/local/bin/gfortran -Wall -g -ffixed-form -fno-second-underscore -arch x86_64 -fPIC -O3 -funroll-loops
Fortran f90 compiler: gfortran -g -fdefault-real-8 -ffixed-form -O2 -fPIC -arch x86_64 -fPIC -O3 -funroll-loops
Fortran fix compiler: gfortran -Wall -g -ffixed-form -fno-second-underscore -g -fdefault-real-8 -ffixed-form -O2 -fPIC -arch x86_64 -fPIC -O3 -funroll-loops
compile options: '-I. -I/Users/ppoem/miniconda3/envs/py3/lib/python3.7/site-packages/netcdf-shared/include -I/var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/src.macosx-10.9-x86_64-3.7 -I/Users/ppoem/miniconda3/envs/py3/lib/python3.7/site-packages/numpy/core/include -I/Users/ppoem/miniconda3/envs/py3/include/python3.7m -c'
gfortran:fix: ./pyscrip.f90
/usr/local/bin/gfortran -Wall -g -arch x86_64 -Wall -g -undefined dynamic_lookup -bundle /var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/src.macosx-10.9-x86_64-3.7/scripmodule.o /var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/src.macosx-10.9-x86_64-3.7/fortranobject.o /var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/pyscrip.o kinds_mod.o constants.o iounits.o netcdf.o grids.o remap_vars.o remap_distwgt.o remap_conserv.o remap_bilinear.o remap_bicubic.o remap_read.o remap.o timers.o remap_write.o -L. -L/Users/ppoem/miniconda3/envs/py3/lib/python3.7/site-packages/netcdf-shared/lib -L/usr/local/Cellar/gcc/9.2.0_1/lib/gcc/9/gcc/x86_64-apple-darwin18/9.2.0 -L/usr/local/Cellar/gcc/9.2.0_1/lib/gcc/9/gcc/x86_64-apple-darwin18/9.2.0/../../.. -L/usr/local/Cellar/gcc/9.2.0_1/lib/gcc/9/gcc/x86_64-apple-darwin18/9.2.0/../../.. -lnetcdf -lnetcdff -ljpeg -lmfhdf -ldf -lhdf5 -lhdf5_fortran -lhdf5_hl -lhdf5hl_fortran -lgfortran -o ./scrip.cpython-37m-darwin.so
ld: library not found for -ljpeg
collect2: error: ld returned 1 exit status

error: Command "/usr/local/bin/gfortran -Wall -g -arch x86_64 -Wall -g -undefined dynamic_lookup -bundle /var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/src.macosx-10.9-x86_64-3.7/scripmodule.o /var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/src.macosx-10.9-x86_64-3.7/fortranobject.o /var/folders/7_/gjm2km6n4517h3ycz8mvk6qw0000gn/T/tmpy520zimy/pyscrip.o kinds_mod.o constants.o iounits.o netcdf.o grids.o remap_vars.o remap_distwgt.o remap_conserv.o remap_bilinear.o remap_bicubic.o remap_read.o remap.o timers.o remap_write.o -L. -L/Users/ppoem/miniconda3/envs/py3/lib/python3.7/site-packages/netcdf-shared/lib -L/usr/local/Cellar/gcc/9.2.0_1/lib/gcc/9/gcc/x86_64-apple-darwin18/9.2.0 -L/usr/local/Cellar/gcc/9.2.0_1/lib/gcc/9/gcc/x86_64-apple-darwin18/9.2.0/../../.. -L/usr/local/Cellar/gcc/9.2.0_1/lib/gcc/9/gcc/x86_64-apple-darwin18/9.2.0/../../.. -lnetcdf -lnetcdff -ljpeg -lmfhdf -ldf -lhdf5 -lhdf5_fortran -lhdf5_hl -lhdf5hl_fortran -lgfortran -o ./scrip.cpython-37m-darwin.so" failed with exit status 1

make: *** [scrip_ext] Error 1

In Python, when I import pyroms, I still can't import it successfully.

Code: Select all

>>> import pyroms
WARNING:root: scrip could not be imported. Remapping functions will not be available
>>> 

Could anybody suggest what does the error I got mean and how should I solve it please?

Another thing is that, I'm unsure if I installed all netCDF packages correctly. I installed netCDF for python using conda-forge or pip. I built netcdf-c and netcdf-fortran manually in a different folder but modified PYTHONPATH to link to it. Are all the netcdf packages supposed to be in the same place? (It works when I run ROMS on this computer (macOS 10.14.6, Python 3.7)

Any help will be highly appreciated.

Best,
Pad

[kate]

However you try installing things, try installing the jpeg library:

Code: Select all

ld: library not found for -ljpeg

The pyroms message about not finding scrip is just a warning. If you don't want to be remapping things with scrip, it is safe to ignore it.

[Phadtaya]

Dear Kate,

Thank you very much for your help. When I look up about the jpeg library, I found that there is a python image library (Pillow package) already installed. I'm not sure if it is the same kind of jpeg library you meant? I then searched for "ld: library not found for -ljpeg" and found that the issue might be related to the path setting or linking. I'm still trying to solve it (any further advice will be also appreciated), but glad to hear that the warning could be ignored when the remapping function is not being used.

Thank you,
Pad

[m.hadfield]

Are you uysing Anaconda? are you able to? The latest instructions assume that that's what you are using. Anaconda handles all the netCDF dependencies automatically?

[Phadtaya]

I'm using Python via miniconda with netcdf installation via conda-forge or pip install. I think that this Python's netCDF package is netCDF-Python while netCDF-C and -Fortran should be installed separately? Please correct if I'm wrong. (I just start using ROMS and now run it on my laptop. I couldn't make it run before I manually build netCDF-C and netcdf-Fortran.)

[m.hadfield]

The guidelines if you're running the Conda package manager:

• You can use PIP to install modules into your Conda environment, but do this only when you have to. You don't have to for the netCDF library or any of its dependencies. The reason for avoiding PIP with Conda is that you are more likely to get duplicates, which lead to strange results.
• Install all the libraries and executables you can in your Conda environment, not in the system.

The PyROMS readme file lists the following prerequisites, available from Conda-Forge:

• Python >= 3.4 (Python 3.7 currently recommended for new environments)
• numpy
• scipy
• matplotlib
• basemap
• netcdf4
• cftime
• lpsolve55
• pip

I think that if you install all those then the JPEG library will be installed also, as a dependency of the HDF5 library which is a dependency of the netCDF-4 library (netcdf4 package). But maybe my copy of the JPEG library got installed via a some other route. But I haven't checked this out. What I need to do is create a new, minimal Conda environment, install only the above packages, and check that everything builds OK.

The fact that you've built netCDF-C and -Fortran libraries separately in order to run ROMS shouldn't affect your Conda environment, provided you don't have environment variables set up that are confusing things. You don't want to mix the Conda and external libraries.

Actually, you can build ROMS with Conda's netCDF system, but this is complicated by the fact that in the Conda system the nf-config command is broken, so all the libraries (currently 9 of them) have to be specified explicitly, as in the PyROMS SCRIP makefile. See line 42 of that file: I constructed that by trial and error.

Who'd be a scientific software developer?

[Phadtaya]

Hello Mark,

Thank you so much for the super helpful guideline! It helps me understand how things work better. I have created a Python 3.7 environment in order to run Pyroms, but I think I might mixed things up when I was trying to set path variables to link to libraries. I will try again with a new environment.

[carlosag]

Hello everybody,
I have the same problem. I followed the instructions in
https://github.com/ESMG/pyroms

and did
$ conda info | grep "active env location"
$ export PREFIX=/to/my/path
$ make DEVELOP=1 PREFIX=$PREFIX install
$ mv -vf scrip*.so ../../../pyroms

but still I get
>>> import pyroms
WARNING:root: scrip could not be imported. Remapping functions will not be available


I am using anaconda and python 3.7.6. I did have troubles installing pyroms with a newer python.
Can someone please give some light to this?
Thank you in advance.
Carlosag

[kate]

It has been a while since I could get scrip to work. The solution is to give up on it, switch to remapping with ESMPy. There's an example using xesmf in examples/Arctic_HYCOM_GLBy.

[carlosag]

Hello,
I solved it. It was an exportation problem. However, now it works. Thank you all.
Best regards.