gmake

gmake

This article first appeared in the HPC Newsletter. It as been revised and updated to more closely match the ROMS makefile.

Over the years, the community has moved from the stance of writing portable Makefiles to a stance of just using a powerful, portable make. The make section described a portable subset of make features. We now delve into some of the more powerful tools available in GNU make. See also Managing projects with GNU Make by Robert Mecklenburg, 2005.

Pattern Rules

The core of make hasn't changed in decades, but concentrating on gmake allows one to make use of its nifty little extras designed by real programmers to help with real projects. The first change that matters to my Makefiles is change from suffix rules to pattern rules. I've always found the .SUFFIXES list to be odd, especially since .f90 is not in the default list. Good riddance to all of that! For a concrete example, the old way to provide a rule for going from file.f90 to file.o is:

 .SUFFIXES: .o .f90 .F .F90

 .f90.o:
  <TAB> $(FC) -c $(FFLAGS) $<

while the new way is:

 %.o: %.f90
  <TAB> $(FC) -c $(FFLAGS) $<

In fact, going to a uniform make means that we can figure out what symbols are defined and use their standard values - in this case, $(FC) and $(FFLAGS) are the built-in default names for the compiler and its options. If you have any questions about this, you can always run make with the -p (or --print-data-base) option. This prints out all the rules make knows about, such as:

 # default

 COMPILE.f = $(FC) $(FFLAGS) $(TARGET_ARCH) -c

Printing the rules database will show variables that make is picking up from the environment, from the Makefile, and from its built-in rules - and which of these sources is providing each value.

Assignments

In the old days, I only used one kind of assignment: = (equals sign). Gmake has several kinds of assignment (other makes might as well, but I no longer have to know or care). An example of the power of GNU make is shown by an example from my Cray X1 Makefile. There is a routine which runs much more quickly if a short function in another file is inlined. The way to accomplish this is through the -O inlinefrom=file directive to the compiler. I can't add this option to FFLAGS, since the inlined routine won't compile with this directive - it is only the one file that needs it. I had:

 FFLAGS = -O 3,aggress -e I -e m
 FFLAGS2 = -O 3,aggress -O inlinefrom=lmd_wscale.f90 -e I -e m

 lmd_skpp.o:
  <TAB> $(FC) -c $(FFLAGS2) $*.f90

The first change I can make to this using other assignments is:

 FFLAGS := -O 3,aggress -e I -e m
 FFLAGS2 := $(FFLAGS) -O inlinefrom=lmd_wscale.f90

The := assignment means to evaluate the right hand side immediately. In this case, there is no reason not to, as long as the second assignment follows the first one (since it's using the value of $(FFLAGS). For the plain equals, make doesn't evaluate the right-hand side until its second pass through the Makefile. However, gmake supports an assignment that avoids the need for FFLAGS2 entirely:

 lmd_skpp.o: FFLAGS += -O inlinefrom=lmd_wscale.f90

What this means is that for the target lmd_skpp.o only, append the inlining directive to FFLAGS. I think this is pretty cool!

One last kind of assignment is to set the value only if there is no value from somewhere else (the environment, for instance):

       FC ?= mpxlf90_r

If we use := or =, we would override the value from the environment.

Include and a Few Functions

One reasonably portable make feature is the include directive. It can be used to clean up the Makefile by putting bits in an include file. The syntax is simply:

 include file

and we use it liberally to keep the project information neat. We can also include a file with the system/compiler information in it, assuming we have some way of deciding which file to include. We can use uname -s to find out which operating system we're using. We also need to know which compiler we're using.

One user-defined variable is called FORT, the name of the Fortran compiler. This value is combined with the result of uname -s to provide a machine and compiler combination. For instance, ftn on Linux is the Cray cross-compiler. This would link to a different copy of the NetCDF library and use different compiler flags than the Intel compiler which might be on the same system.

 # The user sets Fortran Compiler:

    FORT ?= ftn

      OS := $(shell uname -s| sed 's/[\/ ]/-/g'))

 include $(COMPILERS)/$(OS)-$(strip $(FORT)).mk

We pick one include file at compile time, here picking Linux-ftn.mk, containing the Cray cross-compiler information. The value Linux comes from the uname command, the ftn comes from the user, and the two are concatenated. The sed command will turn the slash in UNICOS/mp into a dash; the native Cray include file is UNICOS-mp-ftn.mk. Strip is a built-in function to strip away any extra white space.

Another tricky system is CYGWIN, which puts a version number in the uname output, such as CYGWIN_NT-5.1. GNU make has quite a few built-in functions plus allows the user to define their own functions. One of the built-in functions allows us to do string substitution:

 MACHINE := $(patsubst CYGWIN_%,CYGWIN,$(MACHINE))

In make, the % symbol is a sort of wild card, much like * in the shell. Here, we match CYGWIN followed by an underscore and anything else, replacing the whole with simply CYGWIN. Another example of a built-in function is the substitution we do in:

 objects = $(subst .F,.o,$(sources))

where we build our list of objects from the list of sources. There are quite a few other functions, plus the user can defined their own. From the book:

GNU make supports both built-in and user-defined functions.

A function invocation looks much like a variable reference, but

includes one or more parameters separated by commas. Most built-in

functions expand to some value that is then assigned to a variable

or passed to a subshell. A user-defined function is stored in a

variable or macro and expects one or more parameters to be passed

by the caller.

We will show some user-defined functions below.

Conditionals

We used to have way too many Makefiles, a separate one for each of the serial/MPI/OpenMP versions on each system (if supported). For instance, the name of the IBM compiler changes when using MPI; the options change for OpenMP. The compiler options also change when using 64-bit addressing or for debugging, which we were manually setting. A better way to do this is to have the user select 64-bit or not, MPI or not, etc, then use conditionals. A complete list of the user definable make variables is given in makefile.

GNU make supports two kinds of if test, ifdef and ifeq (plus the negative versions ifndef, ifneq). The example from the book is:

 ifdef COMSPEC
 #  We are running Windows
 else
 #  We are not on Windows
 endif

An example from the IBM include file is:

 ifdef USE_DEBUG
   FFLAGS += -g -qfullpath
 else
   FFLAGS += -O3 -qstrict
 endif

To test for equality, an example is:

 ifeq ($(USE_MPI),on)
 #  Do MPI things
 endif

or

 ifeq "$(USE_MPI)" "on"
 #  Do MPI things
 endif

The user has to set values for the USE_MPI, USE_OPENMP, USE_DEBUG, and USE_LARGE switches in the Makefile before the compiler-dependent piece is included:

    USE_MPI ?= on
 USE_OPENMP ?=
  USE_DEBUG ?=
  USE_LARGE ?= on

The Makefile uses the conditional assign ?= in case a build script is used to set them in the environment. Be sure to leave the switches meant to be off set to an empty string - the string off will test true on an ifdef test.

Multiple Source Directories the ROMS Way

There's more than one way to divide your sources into separate directories. The choices we have made include non-recursive make and putting the temporary files in their own $(BUILD_DIR) directory. These include the .f90 files which have been through the C preprocessor, object files, module files, and libraries.

Directory Structure

The directory structure of the source code has the top directory, a Master directory, a ROMS directory with a number of subdirectories, and several other directories. Master contains the main program while the rest contain sources for libraries and other files. Note that the bulk of the source code gets compiled into files that become libraries with the ar command, one library per directory. There is also a Compilers directory for the system- and compiler-specific Makefile components.

Conditionally Including Components

The makefile will build the lists of libraries to create and source files to compile. They start out empty at the top of the makefile:

 sources :=
 libraries :=

That's simple enough, but the list of directories to search for these sources will depend on the options chosen by the user, not just in the make options, but inside the ROMS_HEADER file as well. How does this happen? Once make knows how to find the ROMS_HEADER, it is used by cpp to generate an include file telling make about these other options.

 MAKE_MACROS := Compilers/make_macros.mk
 MACROS := $(shell cpp -P $(ROMS_CPPFLAGS) Compilers/make_macros.h > \
                  $(MAKE_MACROS); $(CLEAN) $(MAKE_MACROS))

The make_macros.h file contains blocks such as:

 #ifdef SWAN_COUPLING
   USE_SWAN := on
 #else
   USE_SWAN :=
 #endif

The resulting make_macros.mk file will simply end up with either

 USE_SWAN := on

or

 USE_SWAN :=

This file can then be included by the makefile and the variable USE_SWAN will have the correct state for this particular compilation. We can now use it and all the similar flags to build a list of directories.

We initialize two lists:

 modules  :=
 includes :=    ROMS/Include

Add the adjoint bits:

 ifdef USE_ADJOINT
   modules  +=    ROMS/Adjoint
 endif
 ifdef USE_ADJOINT
   includes +=    ROMS/Adjoint
 endif

Add the bits we'll always need:

 modules  +=    ROMS/Nonlinear \
                ROMS/Functionals \
                ROMS/Utility \ 
                ROMS/Modules
 includes +=    ROMS/Nonlinear \
                ROMS/Utility \
                ROMS/Drivers

Then we add in some more:

 ifdef USE_SWAN
  modules  +=    Waves/SWAN/Src
  includes +=    Waves/SWAN/Src
 endif

 modules  +=    Master
 includes +=    Master Compilers

Now that our lists are complete, let's put them to use:

 vpath %.F $(modules)
 vpath %.h $(includes)
 vpath %.f90 $(BUILD_DIR)
 vpath %.o $(BUILD_DIR)

 include $(addsuffix /Module.mk,$(modules))

 CPPFLAGS += $(patsubst %,-I%,$(includes))

1. vpath is a standard make feature for providing a list of directories for make to search for files of different types. Here we are saying that *.F files can be found in the directories provided in the $(modules) list, and so on for the others.
2. For each directory in the $(modules) list, make will include the file Module.mk that is found there. More on these later.
3. For each directory in the $(includes) list, add that directory to the list searched by cpp with the -I flag.

User-defined make Functions

The Module.mk fragments mentioned before call some user-defined functions. It's time to show these functions and talk about how they work. They get defined in the top Makefile:

 #--------------------------------------------------------------------------
 #  Make functions for putting the temporary files in $(BUILD_DIR)
 #--------------------------------------------------------------------------

 # $(call source-dir-to-binary-dir, directory-list)
 source-dir-to-binary-dir = $(addprefix $(BUILD_DIR)/, $(notdir $1))

 # $(call source-to-object, source-file-list)
 source-to-object = $(call source-dir-to-binary-dir,   \
                  $(subst .F,.o,$(filter %.F,$1)))

 # $(call f90-source, source-file-list)
 f90-source = $(call source-dir-to-binary-dir,     \
                  $(subst .F,.f90,$1))

 # $(call make-library, library-name, source-file-list)
 define make-library
    libraries += $(BUILD_DIR)/$1
    sources   += $2

    $(BUILD_DIR)/$1: $(call source-dir-to-binary-dir,    \
                     $(subst .F,.o,$2))
        $(AR) $(ARFLAGS) $$@ $$^
        $(RANLIB) $$@
 endef

 # $(call one-compile-rule, binary-file, f90-file, source-files)
 define one-compile-rule
   $1: $2 $3
        cd $$(BUILD_DIR); $$(FC) -c $$(FFLAGS) $(notdir $2)

   $2: $3
        $$(CPP) $$(CPPFLAGS) $$(MY_CPP_FLAGS) $$< > $$@
        $$(CLEAN) $$@
 endef

 # $(compile-rules)
 define compile-rules
   $(foreach f, $(local_src),       \
     $(call one-compile-rule,$(call source-to-object,$f), \
     $(call f90-source,$f),$f))
 endef

1. We define a function to convert the path from the source directory to the Build directory, called source-dir-to-binary-dir. Note that the Build directory is called $(BUILD_DIR) here. All it does is strip off the leading directory with the the built-in function notdir, then paste on the \code{Build} directory.
2. Next comes source-to-object, which calls the function above to return the object filename when given the source filename. It assumes that all sources have a .F extension.
3. A similar function is f90-source, which returns the name of the intermediate source which is created by cpp from our .F file.
4. The Module.mk fragment in each library source directory invokes make-library, which takes the library name and the list of sources as its arguments. The function adds its library to the global list of libraries and provides rules for building itself. The double dollar signs are to delay the variable substitution. Note that we call source-dir-to-binary-dir instead of source-to-object - this is a work-around for a make bug.
5. The next, one-compile-rule, takes three arguments: the .o filename, the .f90 filename, and the .F filename. From these, it produces the make rules for running cpp and the compiler.
   
   A note on directories: make uses vpath to find the source file where it resides. It would be possible to compile from the top directory and put the .o file in Build with the appropriate arguments, but I don't know how to get the .mod file into Build short of a mv command. Likewise, if we compile in the top directory, we need to know the compiler option to tell it to look in Build for the .mod files it uses. Doing a cd to Build before compiling is just simpler.
6. The last, compile-rules, is given a list of sources, then calls one-compile-rule once per source file.

Again, you can invoke make -p to see how make internally transforms all this into actual targets and rules.

Library Module.mk

In each library directory, there is a file called Module.mk which gets included by the top level makefile. These Module.mk bits build onto the list of sources and libraries to be compiled and built, respectively. These Module.mk files look something like:

 local_sub  := ROMS/Nonlinear 
 local_lib  := libNLM.a

 local_src  := $(wildcard $(local_sub)/*.F)

 $(eval $(call make-library,$(local_lib),$(local_src)))

 $(eval $(compile-rules))

First, we provide the name of the current directory and the library to be built from the resident sources. Next, we use the wildcard function to search the subdirectory for these sources. Note that every .F file found will be compiled. If you have half-baked files that you don't want used, make sure they have a different extension.

Each subdirectory is resetting the local_src variable. That's OK because we're saving the values in the global sources variable inside the make-library function, which also adds the local library to the libraries list. The compile-rules function uses this local_src variable to generate the rules for compiling each file, placing the resulting files in the Build directory.

Main Program

The main program is in a directory called Master and its Module.mk is similar to the library one:

 local_sub  := Master
 local_src  := $(wildcard $(local_sub)/*.F)

 local_objs := $(subst .F,.o,$(local_src))
 local_objs := $(addprefix $(BUILD_DIR)/, $(notdir $(local_objs)))

 sources    += $(local_src)

 ifdef LD_WINDOWS
 $(BIN): $(libraries) $(local_objs)
         $(LD) $(FFLAGS) $(local_objs) -o $@ $(libraries) $(LIBS_WIN32) $(LDFLAGS)
 else
 $(BIN): $(libraries) $(local_objs)
         $(LD) $(FFLAGS) $(LDFLAGS) $(local_objs) -o $@ $(libraries) $(LIBS)
 endif

 $(eval $(compile-rules))

Instead of a rule for building a library, we have a rule for building a binary. In this case, the name of the binary is defined elsewhere. The binary depends on the libraries getting compiled first, as well as the local sources. During the link, the $(libraries) are compiled from the sources in the other directories, while $(LIBS) are external libraries such as NetCDF.

Top Level Makefile

Now we get to the glue that holds it all together. We've covered many things so far, but there's still a few bits which might be confusing:

1. There can be rare cases where you might have special code for some systems. You can check which system you are on in the .F file with:
   #ifdef X86_64
   ! special stuff
   #endif
   To be sure this is defined on each \code{X86\_64} system, it has to be passed to cpp:
   CPPFLAGS += -D$(shell echo ${OS} | tr "-" "_" | tr [a-z] [A-Z])
   CPPFLAGS += -D$(shell echo ${CPU} | tr "-" "_" | tr [a-z] [A-Z])
   CPPFLAGS += -D$(shell echo ${FORT} | tr "-" "_" | tr [a-z] [A-Z])
   
   CPPFLAGS += -D'ROOT_DIR="$(ROOTDIR)"'
   ifdef ROMS_APPLICATION
   CPPFLAGS += $(ROMS_CPPFLAGS)
   CPPFLAGS += -DNestedGrids=$(NestedGrids)
   MDEPFLAGS += -DROMS_HEADER="$(HEADER)"
   endif
   This guarantees that CPPFLAGS will have terms in it such as:
   -DLINUX -DX86_64 -DPGI
   -D'ROOT_DIR="/export/staffdata/kate/roms/trunk"' -DSHOREFACE
   -D'HEADER="shoreface.h"' -D'ROMS_HEADER="shoreface.h"'
   -DNestedGrids=1
2. For mod_strings.F, there is a special case:
   $(BUILD_DIR)/mod_strings.f90: CPPFLAGS += -DMY_OS='"$(OS)"' \
   -DMY_CPU='"$(CPU)"' -DMY_FORT='"$(FORT)"' \
   -DMY_FC='"$(FC)"' -DMY_FFLAGS='"$(FFLAGS)"'
   allowing ROMS to report in its output:
   Operating system : Linux
   CPU/hardware  : x86_64
   Compiler system  : pgi
   Compiler command : pgf90
   Compiler flags  : -O3 -tp k8-64 -Mfree
   
   Local Root  : /export/staffdata/kate/roms/trunk
   Header Dir  : ./ROMS/Include
   Header file  : shoreface.h
   Though this doesn't seem to work on the Mac.
3. The very first makefile I showed had a set list of files to remove on make clean. We now build a list, called clean_list:
   clean_list := core *.ipo $(BUILD_DIR)
   
   ifeq "$(strip $(BUILD_DIR))" "."
   clean_list := core *.o *.oo *.mod *.f90 lib*.a *.bak
   clean_list += $(CURDIR)/*.ipo
   endif
   In other words, we want to clean up the Build directory unless it happens to be the top level directory, in which case we only want to remove specific files there.
4. all is the first target that gets seen by make, making it the default target. In this case, we know there is only the one binary, whose name we know - the book shows what to do with more than one binary. Both all and clean are phony targets in that no files of those names get generated - make has the .PHONY designation for such targets. Also, the clean target doesn't require any compiler information, so the compiler include doesn't happen if the target is clean:
   ifneq "$(MAKECMDGOALS)" "clean"
   include $(COMPILERS)/$(OS)-$(strip $(FORT)).mk
   endif
   $(MAKECMDGOALS) is a special variable containing the current make target.
5. We'll be creating different executable names, depending on which options we've picked:
   BIN := $(BINDIR)/romsS
   ifdef USE_DEBUG
   BIN := $(BINDIR)/romsG
   else
   ifdef USE_MPI
   BIN := $(BINDIR)/romsM
   endif
   ifdef USE_OpenMP
   BIN := $(BINDIR)/romsO
   endif
   endif
   Note: In SVN revision 933 (January 26, 2019) all ocean_*.in files were renamed to roms_*.in and all ocean* ROMS executables were renamed to roms* in order to facilitate and clarify model coupling efforts. More information can be found in the ROMS repository Trac ticket #794. If you are working with a ROMS release prior to revision 933, you will find that romsS, romsM, and romsO will read oceanS, oceanG, oceanM, and oceanO instead.
   
   
6. The NetCDF library gets included during the final link stage. However, we are now using the Fortran 90 version of it which requires its module information as well. We just copy the .mod files into the Build directory:
   NETCDF_MODFILE := netcdf.mod
   TYPESIZES_MODFILE := typesizes.mod
   
   $(BUILD_DIR)/$(NETCDF_MODFILE): | $(BUILD_DIR)
   cp -f $(NETCDF_INCDIR)/$(NETCDF_MODFILE) $(BUILD_DIR)
   
   $(BUILD_DIR)/$(TYPESIZES_MODFILE): | $(BUILD_DIR)
   cp -f $(NETCDF_INCDIR)/$(TYPESIZES_MODFILE) $(BUILD_DIR)
   Old versions of NetCDF do not have the typesizes.mod file, in which case it has to be removed from the following dependency list:
   $(BUILD_DIR)/MakeDepend: makefile \
   $(BUILD_DIR)/$(NETCDF_MODFILE) \
   $(BUILD_DIR)/$(TYPESIZES_MODFILE) \
   | $(BUILD_DIR)

1. Then there is MakeDepend itself. This file is created by the Perl script sfmakedepend. MakeDepend gets created by make depend and included on make's second pass through the makefile:
   depend: $(BUILD_DIR)
   $(SFMAKEDEPEND) $(MDEPFLAGS) $(sources) > $(BUILD_DIR)/MakeDepend
   
   ifneq "$(MAKECMDGOALS)" "clean"
   -include $(BUILD_DIR)/MakeDepend
   endif
   The dash before the include tells make to ignore errors so that make depend will succeed before the file exists. The MakeDepend file will contain the include and module dependencies for each source file, such as:
   Build/mod_diags.o: tile.h cppdefs.h globaldefs.h shoreface.h
   Build/mod_diags.f90: tile.h cppdefs.h globaldefs.h shoreface.h
   Build/mod_diags.o: Build/mod_kinds.o Build/mod_param.o Build/mod_diags.f90
   Note that the .h files are included by cpp, so that both the .f90 and .o files become out of date when an include file is modified. Without the module dependencies, make would try to build the sources in the wrong order and the compiler would fail with a complaint about not finding mod_param.mod, for instance.

Final Warnings

The cost of this nifty make stuff is:

1. We're a little closer to the gnu make bugs here, and we need a newer version of gnu make than before (version 3.81, 3.80 if you're lucky). Hence this stuff at the top of the makefile:
   NEED_VERSION := 3.80 3.81
   $(if $(filter $(MAKE_VERSION),$(NEED_VERSION)),, \
   $(error This makefile requires one of GNU make version $(NEED_VERSION).))
2. The Makefile dependencies get just a little trickier every change we make. Note that F90 has potentially both include and module use dependencies. The book example uses the C compiler to produce its own dependencies for each source file into a corresponding .d file to be included by make. Our Fortran compilers are not so smart. For these hairy compiles, it's critical to have accurate dependency information unless we're willing to make clean between compiles.