Missing openmpi directory during compiling

[JMette]

Hi all,

I have a question regarding the configuration of the upwelling test case with openmpi on our computing cluster. When I run in serial, everything works fine. Also, on my private computer, I'm able to run it with mpi in the default config. When I use openmpi, during compile time I get the following output:

Code: Select all

f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow -ffree-form -ffree-line-length-none mod_ice.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow mod_stepping.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow -ffree-form -ffree-line-length-none round.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow mod_eoscoef.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow lapack_mod.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow lubksb.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow ludcmp.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow mp_routines.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow read_icepar.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow sqlq.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow mod_param.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow mod_iounits.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow nrutil.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow mod_scalars.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow -ffree-form -ffree-line-length-none mod_sediment.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow mod_behavior.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow shapiro.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow gasdev.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow -fno-strict-overflow ran_state.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow obs_k2z.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow ran1.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow exchange_2d.f90
cd /home/jmette/roms_project/roms_test/upwelling/Build_romsG; /usr/bin/mpif90 -c -frepack-arrays -fallow-argument-mismatch        -g -O0 -fbounds-check -fbacktrace -fcheck=all -finit-real=nan -ffpe-trap=invalid,zero,overflow mod_parallel.f90
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
f951: Warning: Nonexistent include directory ‘/usr/lib/x86_64-linux-gnu/openmpi/lib/../../fortran/gfortran-mod-15/openmpi’ [-Wmissing-include-dirs]
mod_parallel.f90:19: Error: Can't open included file 'mpif.h'
make: *** [ROMS/Modules/Module.mk:15: /home/jmette/roms_project/roms_test/upwelling/Build_romsG/mod_parallel.o] Error 1
make: *** Waiting for unfinished jobs....

For what I understand from that error, the build script cannot find the openmpi installation, right? I wait right now for more information on that from the person responsible for the computation cluster. But then would be my question:

How do I specify the custom path in the build script? Is it this block and if yes, how would a working example look like (I'm not so experienced with bash usage & co)?

Code: Select all

#--------------------------------------------------------------------------
# If applicable, use my specified library paths.
#--------------------------------------------------------------------------

 export USE_MY_LIBS=no            # use system default library paths
#export USE_MY_LIBS=yes           # use my customized library paths

MY_PATHS=${COMPILERS}/my_build_paths.sh

if [ "${USE_MY_LIBS}" == "yes" ]; then
  source ${MY_PATHS} ${MY_PATHS}
fi

Attached is also my build script, I'm very glad for any help!

[wilkin]

ROMS should be able to find what it needs in the system defaults, so your sysadmin build of MPI might be problematic.

If you need to customize the paths, it's best to do it in ./Compilers/my_build_paths.csh which will be used when (in build_roms.sh) you activate ...

Code: Select all

export USE_MY_LIBS=yes           # use my customized library paths
MY_PATHS=${COMPILERS}/my_build_paths.sh

[JMette]

Thanks for the help. I also found now the corresponding wiki pages.
But the follow-up problem is, that the default paths for all libraries in the my_build_paths.sh are (understandably) not correct and it then cannot find e.g. the netcdf-library. If I understand it correctly, I have do define the paths for every library by hand in this script? Or is there a way to use only a custom path for the mpi installation and keep the other ones from the none-custom build script?

[JMette]

After some correspondence with John Wilkin & David Robertson I tried a few things and achieved some progress:

- I tried to

Code: Select all

export PATH=/path/to/mpi/bin:$PATH

but that also didn't worked out.

- I tried to compile a simple fortran program with mpf90 and that gave the same error message as above, so probably the openmpi-installation on the computing cluster has some flaws.

- With a new, additional openmpi-installation in the home directory of my user and after exporting the new "/openmpi/bin"-path, I was able to compile and run the fortran test program with mpi sucessfully as well as building the upwelling test case. For the test case I get the warning:

Code: Select all

distribute.f90:4179:26:

 4179 |       CALL mpi_allreduce (Asend, Areduce, Npts, MP_FLOAT, handle,       &
      |                          1
......
 4412 |         CALL mpi_allreduce (Asend, Areduce, Isize,                      &
      |                            2
Warning: Rank mismatch between actual argument at (1) and actual argument at (2) (rank-1 and scalar)
distribute.f90:4066:28:

 4066 |         CALL mpi_allreduce (Asend(i), Areduce(i), 1, MPI_INTEGER,       &
      |                            1
......
 4412 |         CALL mpi_allreduce (Asend, Areduce, Isize,                      &
      |                            2
Warning: Type mismatch between actual argument at (1) and actual argument at (2) (INTEGER(4)/REAL(8)).
distribute.f90:3837:26:

 3837 |       CALL mpi_allgather (my_bounds, np, MPI_INTEGER,                   &
      |                          1
......
 5588 |       CALL mpi_allgather (Asend, Npts, MP_FLOAT, Arecv, Npts, MP_FLOAT, &
      |                          2
Warning: Type mismatch between actual argument at (1) and actual argument at (2) (INTEGER(4)/REAL(8)).
distribute.f90:1408:22:

 1408 |       CALL mpi_bcast (ibuffer, Npts, MPI_INTEGER, MyMaster,             &
      |                      1
......
 5036 |       CALL mpi_bcast (A, Asize, MP_FLOAT, MyMaster, OCN_COMM_WORLD,     &
      |                      2
Warning: Type mismatch between actual argument at (1) and actual argument at (2) (INTEGER(4)/REAL(8)).
distribute.f90:218:22:

  218 |       CALL mpi_bcast (A, Npts, MP_FLOAT, MyMaster, MyCOMM, MyError)
      |                      1
......
 5036 |       CALL mpi_bcast (A, Asize, MP_FLOAT, MyMaster, OCN_COMM_WORLD,     &
      |                      2
Warning: Rank mismatch between actual argument at (1) and actual argument at (2) (rank-1 and scalar)

- But when I run the upwelling test case, it gives the following error:

Code: Select all

[*** The MPI_Comm_f2c() function was called before MPI_INIT was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.
[kcn0:1779501] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
--------------------------------------------------------------------------
prterun detected that one or more processes exited with non-zero status,
thus causing the job to be terminated. The first process to do so was:

   Process name: [prterun-kcn0-1779497@1,1] Exit code:    14
--------------------------------------------------------------------------

As far as I found out, this error is caused during the linking process when components from different mpi versions get mixed up, so a function gets called before initialization.

- I then also exported

Code: Select all

export LD_LIBRARY_PATH=/path/to/my/openmpi/lib:$LD_LIBRARY_PATH

which didn't help. So there is probably still some environment variable, which is not configured properly and still points to the old openmpi-installation.

[arango]

Yes, those MPI warnings are typical from gfortran. The MPI libraries are written in C and C++, allowing generic calls to enable polymorphism and type-bound objects in the argument type, calling the parallel communications function. There are specific compiling flags in gfortran for surprising such warnings, but I am very used to them.

[JMette]

You probably meant "suppressing", right? Does these warnings, by any chance can give hint at what point my configuration is incorrect?

For completeness, attached is the full output of ./build_roms.sh. The ROMS source code is under "/home/jmette/roms/" and the new open-mpi installation under "/home/jmette/openmpi/"
If someone spots something, that could help me with the right configuration I would be really thankful.

[robertson]

Are you certain that the 'mpirun' you used is at /home/jmette/openmpi/bin?