gfortran -ffast-math experience

[jcwarner]

Dear all-

I want to strongly encourage removal of gfortran compiler flag: -ffast-math

My recent experience shows this compiler option is not a good choice. The documentation is stating:
[https://gcc.gnu.org/wiki/FloatingPointMath]
" ... the -ffast-math flag which is a shortcut for several options, presenting the least conforming but fastest math mode. It enables -fno-trapping-math, -funsafe-math-optimizations, ...."

Further the "-funsafe-math-optimizations" says "... enables optimizations that allow arbitrary reassociations and transformations with no accuracy guarantees."

This has been a bad experience for me that past few weeks. I have been trying to get a N-S periodic test to run, but it would not maintain periodicity. I finally removed this -ffast-math flag, recompiled, and now all is well.

This is reminiscent of the ifort -fprecise flag, but that is a different story.

In summary: I highly recommend that if you use gfortran you consider to remove the -ffast-math flag from Compilers:
Compilers/Linux-g95.mk:94: FFLAGS += -O3 -ffast-math
Compilers/Darwin-gfortran.mk:92:# FFLAGS += -O3 -ffast-math
Compilers/MINGW-gfortran.mk:81: FFLAGS += -O3 -ffast-math
Compilers/Linux-path.mk:104:# FFLAGS += -O3 -OPT:Ofast -fno-math-errno -ffast-math
Compilers/Linux-gfortran.mk:91: FFLAGS += -O3 -ffast-math
Compilers/CYGWIN-g95.mk:87: FFLAGS += -O3 -ffast-math
Compilers/CYGWIN-gfortran.mk:99: FFLAGS += -O3 -ffast-math
Compilers/MINGW-g95.mk:79: FFLAGS += -O3 -ffast-math

-john

[andres]

I've once heard a similar comment about the -O3 flag, from the IT manager of a modeling group, he was describing it as "aggressive paralelization".

Any experience between -O3 and -O2 ?

In the last ROMS_AGRIF (3.1.1) and CROCO the default flags for gfortran are

FFLAGS1="-O3 -fdefault-real-8 -fdefault-double-8 -mcmodel=medium"

[okami_zxh]

Hi John,
Which version of gfortran are you using?
Xiaohui

[stlaur]

I accidentally ran ROMS with the default values of Linux-gfortran.mk, meaning ffast-math and O3.
Since I had this unexpected data point in hand, I thought I should conduct a few more tests and then report on this thread, in support of the original post (by John Warner) recommending the removal of ffast-math.

Regarding speed, for an identical configuration except for the compiler setting, I get:

(1) O2 gfortran: 201 simulated days per compute day;
(2) O3 gfortran: 232 simulated days per compute day;
(3) ffast-math O3 gfortran: 281 simulated days per compute day.

So in this configuration, the benefit of ffast-math+O3 over straight O3 is a "21% faster" code.
It's not negligible, but not a gigantic improvement either. (The version of gfortran is 9.4.0.)

In terms of reproducibility, the figure attached shows the maximum temperature along the vertical (excluding the surface mixed layer) computed from a monthly-averaged 3D field, after 15 months of integration (everything identical except compiler setting).
The colorscale is -2dec.C to +2dec.C, and the model domain is a continental slope/shelf in the Southern Ocean. The runs with O2 and O3 (top two panels) look the same in this qualitative comparison.
Focusing on the accidental run (i.e. with ffast-math, bottom panel), ffast-math eventually causes the temperature to be approximately 1 degree C warmer from the other two runs in specific areas (see arrows), which by Antarctic standards is a lot.

A person mentioned on this thread that Croco avoids using ffast-math. I think that's a good idea.