upwelling case paralleling compilation with ftn

[xinwang]

The system suggests me to use 'ftn' and module command.I module netcdf and esmf. The following is my build.sh

set parallel = 0
set clean = 1

while ( ($#argv) > 0 )
switch ($1)
case "-noclean"
shift
set clean = 0
breaksw

case "-j"
shift
set parallel = 1
if (`echo $1 | grep -P '^\d+$'` != "" ) then
set NCPUS = "-j $1"
shift
else
set NCPUS = "-j"
endif
breaksw

case "-*":
echo ""
echo "$0 : Unknown option [ $1 ]"
echo ""
echo "Available Options:"
echo ""
echo "-j [N] Compile in parallel using N CPUs"
echo " omit argument for all avaliable CPUs"
echo "-noclean Do not clean already compiled objects"
echo ""
exit 1
breaksw

endsw
end

setenv ROMS_APPLICATION UPWELLING

setenv MY_ROOT_DIR ${HOME}/roms_projects
setenv MY_PROJECT_DIR ${HOME}/roms_projects/upwelling_ftn_netcdf4_parallel

#setenv MY_ROMS_SRC ${MY_ROOT_DIR}/branches/arango
setenv MY_ROMS_SRC ${HOME}/roms/src


setenv COMPILERS ${MY_ROMS_SRC}/Compilers
#setenv COMPILERS ${HOME}/Compilers

setenv USE_MPI on # distributed-memory parallelism
# setenv USE_MPIF90 on # compile with mpif90 script
#liu change
# setenv which_MPI mpich # compile with MPICH library
setenv which_MPI mpich2 # compile with MPICH2 library
# setenv which_MPI openmpi # compile with OpenMPI library

# setenv USE_OpenMP on # shared-memory parallelism

# setenv FORT ifort
# setenv FORT gfortran
# setenv FORT pgi
setenv FORT ftn

setenv USE_DEBUG on # use Fortran debugging flags
setenv USE_LARGE on # activate 64-bit compilation
setenv USE_NETCDF4 on # compile with NetCDF-4 library
setenv USE_PARALLEL_IO on # Parallel I/O with NetCDF-4/HDF5

## setenv USE_MY_LIBS on # use my library paths below


setenv MY_HEADER_DIR ${MY_PROJECT_DIR}

setenv MY_ANALYTICAL_DIR ${MY_PROJECT_DIR}

# Put the binary to execute in the following directory.

setenv BINDIR ${MY_PROJECT_DIR}

# Put the f90 files in a project specific Build directory to avoid conflict
# with other projects.

setenv SCRATCH_DIR ${MY_PROJECT_DIR}/Build

# Go to the users source directory to compile. The options set above will
# pick up the application-specific code from the appropriate place.

cd ${MY_ROMS_SRC}

# Stop if activating both MPI and OpenMP at the same time.

if ( ${?USE_MPI} & ${?USE_OpenMP} ) then
echo "You cannot activate USE_MPI and USE_OpenMP at the same time!"
exit 1
endif

# Remove build directory.

if ( $clean == 1 ) then
make clean
endif

# Compile (the binary will go to BINDIR set above).

if ( $parallel == 1 ) then
make $NCPUS
else
make
endif

When running this build.sh, there are the following errors.
makefile:238: INCLUDING FILE /ncrc/home2/Xin.Wang/make_macros.mk WHICH CONTAINS APPLICATION-DEPENDENT MAKE DEFINITIONS
makefile:234: INCLUDING FILE /ncrc/home2/Xin.Wang/roms_projects/upwelling_ftn_netcdf4_parallel/Build/make_macros.mk WHICH CONTAINS APPLICATION-DEPENDENT MAKE DEFINITIONS
pgf90-Error-Unknown switch: -e
pgf90-Error-Unknown switch: -e
pgf90-Error-Unknown switch: -G
pgf90-Error-Unknown switch: -free-form
pgf90-Error-Unknown switch: -e
pgf90-Error-Unknown switch: -e
pgf90-Error-Unknown switch: -e
pgf90-Error-Unknown switch: -e
pgf90-Error-Unknown switch: -G
pgf90-Error-Unknown switch: -G
pgf90-Error-Unknown switch: -free-form
pgf90-Error-Unknown switch: -free-form
pgf90-Error-Unknown switch: -free-form
pgf90-Error-Unknown switch: -free-form
make: *** [/ncrc/home2/Xin.Wang/roms_projects/upwelling_ftn_netcdf4_parallel/Build/strings.o] Error 1
make: *** Waiting for unfinished jobs....
make: *** [/ncrc/home2/Xin.Wang/roms_projects/upwelling_ftn_netcdf4_parallel/Build/mod_kinds.o] Error 1
make: *** [/ncrc/home2/Xin.Wang/roms_projects/upwelling_ftn_netcdf4_parallel/Build/mod_strings.o] Error 1

I am wondering that there is no pgf90 in file of build.sh. why are there error about pgf90? Many thanks.

[kate]

I assume you are on a newish Linux Cray? Cray has created a wrapper script called ftn, which is invoking pgi Fortran on your system. ftn is sort of like mpif90 on other systems, a command which invokes the Fortran compiler of your loaded module set and also knows how to find the appropriate mpi and so forth.

There is history here, in that ftn was once the name of Cray's own compiler, then later the ROMS Linux-ftn setup was for a Linux cross-compiler that made executables for a Cray. You will have to change the file Compilers/Linux-ftn.mk for your system to match the newest meaning of "ftn". This includes changing the FFLAGS to match that of Linux-pgi.mk. To make things more confusing, our new Cray has more than one compiler module to pick from, so that loading PrgEnv-pgi points "ftn" to the pgi compiler while loading PrgEnv-cray points it at a Cray compiler:

fish2 1000% module avail PrgEnv

------------------------------- /opt/modulefiles -------------------------------
PrgEnv-cray/4.0.46(default) PrgEnv-pgi/4.0.46(default)
PrgEnv-gnu/4.0.46(default)

Perhaps you should copy over Linux-pgi.mk and call it Linux-ftnpgi.mk, changing the name of the compiler in there to ftn, then set FORT=ftnpgi.

[xinwang]

You are right. This is first time for me to use Linux Cray. Thanks for your reply. Now, I can compile successfully in parallel using ftn, and create an executable file (oceanG). I try to aprun and mpirun, but it is said that 'command not found'. Which command can be used to run executable file? Many thanks !

[kate]

On our Cray, aprun works. Here is my job script for batch jobs:

Code: Select all

#!/bin/bash
#PBS -N NEP6
#PBS -q standard          
#PBS -l walltime=24:00:00 
#PBS -l nodes=1:ppn=12
#PBS -j oe
               
cd $PBS_O_WORKDIR
ulimit -Sc unlimited
export MPICH_ABORT_ON_ERROR=1

NP=$(( $PBS_NUM_NODES * $PBS_NUM_PPN ))
#aprun -n $NP ./oceanG ocean_nep.in
aprun -n $NP ./oceanM ocean_nep.in