MPI time profiling results?

Post ROMS benchmark results

Moderators: arango, robertson

Post Reply
Message
Author
jamak3
Posts: 2
Joined: Tue Sep 28, 2010 4:26 am
Location: California Polytechnic State University

MPI time profiling results?

#1 Unread post by jamak3 »

I'm trying to make sense of the time profiling output for an MPI run of ROMS, with example output below. Is the elapsed time for message passing included in the elapsed time for the non-linear model? For example, does the time elapsed shown for the Model 2D kernel also include the time for 2D halo exchanges? Or do you have to add Model 2D kernel time and 2D halo exchange time to get the total time for 2D stepping?

Nonlinear model elapsed time profile:

Initialization ................................... 478.991 ( 0.1024 %)
Reading of input data ............................ 295.288 ( 0.0631 %)
Processing of input data ......................... 24366.475 ( 5.2101 %)
Processing of output time averaged data .......... 2034.842 ( 0.4351 %)
Computation of vertical boundary conditions ...... 374.829 ( 0.0801 %)
Computation of global information integrals ...... 5998.291 ( 1.2826 %)
Writing of output data ........................... 15187.017 ( 3.2473 %)
Model 2D kernel .................................. 221638.845 (47.3913 %)
2D/3D coupling, vertical metrics ................. 6819.473 ( 1.4582 %)
Omega vertical velocity .......................... 14814.303 ( 3.1676 %)
Equation of state for seawater ................... 14721.860 ( 3.1479 %)
GLS vertical mixing parameterization ............. 74444.107 (15.9178 %)
3D equations right-side terms .................... 3313.498 ( 0.7085 %)
3D equations predictor step ...................... 12071.718 ( 2.5812 %)
Pressure gradient ................................ 4012.504 ( 0.8580 %)
Harmonic mixing of tracers, geopotentials ........ 4142.211 ( 0.8857 %)
Harmonic stress tensor, S-surfaces ............... 1962.384 ( 0.4196 %)
Corrector time-step for 3D momentum .............. 33187.006 ( 7.0961 %)
Corrector time-step for tracers .................. 16517.137 ( 3.5317 %)
Total: 456380.778 97.5844

Nonlinear model message Passage profile:

Message Passage: 2D halo exchanges ............... 198439.255 (42.4308 %)
Message Passage: 3D halo exchanges ............... 79077.268 (16.9085 %)
Message Passage: 4D halo exchanges ............... 29843.343 ( 6.3812 %)
Message Passage: data broadcast .................. 11739.061 ( 2.5101 %)
Message Passage: data reduction .................. 5144.240 ( 1.1000 %)
Message Passage: data gathering .................. 3363.179 ( 0.7191 %)
Message Passage: data scattering.................. 408.240 ( 0.0873 %)
Total: 328014.586 70.1369

All percentages are with respect to total time = 467677.894

hhiester
Posts: 1
Joined: Tue Aug 28, 2012 8:10 pm
Location: Florida State University

Re: MPI time profiling results?

#2 Unread post by hhiester »

I was also trying to understand this question. The profile breakdown looks similar (see below). Is this normal for a ROMS run? The run is on 64 processors and uses serial I/O. In addition, is it possible run times are being hampered by the serial I/O/what kind of speed up would be expected by switching to parallel I/O?

Nonlinear model elapsed time profile:

Initialization ................................... 13643.532 ( 1.2258 %)
OI data assimilation ............................. 11.378 ( 0.0010 %)
Reading of input data ............................ 22709.763 ( 2.0403 %)
Processing of input data ......................... 80238.841 ( 7.2088 %)
Computation of vertical boundary conditions ...... 1864.843 ( 0.1675 %)
Computation of global information integrals ...... 358.941 ( 0.0322 %)
Writing of output data ........................... 16781.614 ( 1.5077 %)
Model 2D kernel .................................. 483542.947 (43.4424 %)
2D/3D coupling, vertical metrics ................. 9626.206 ( 0.8648 %)
Omega vertical velocity .......................... 11943.059 ( 1.0730 %)
Equation of state for seawater ................... 16972.353 ( 1.5248 %)
Atmosphere-Ocean bulk flux parameterization ...... 8381.848 ( 0.7530 %)
My2.5 vertical mixing parameterization ........... 138098.676 (12.4070 %)
3D equations right-side terms .................... 12738.116 ( 1.1444 %)
3D equations predictor step ...................... 49374.785 ( 4.4359 %)
Pressure gradient ................................ 3554.001 ( 0.3193 %)
Harmonic mixing of tracers, isopycnals ........... 13814.746 ( 1.2411 %)
Harmonic stress tensor, geopotentials ............ 29147.193 ( 2.6186 %)
Corrector time-step for 3D momentum .............. 24644.077 ( 2.2141 %)
Corrector time-step for tracers .................. 161828.397 (14.5390 %)
Total: 1099275.315 98.7609

Nonlinear model message Passage profile:

Message Passage: 2D halo exchanges ............... 523336.329 (47.0175 %)
Message Passage: 3D halo exchanges ............... 159054.711 (14.2898 %)
Message Passage: 4D halo exchanges ............... 109670.134 ( 9.8530 %)
Message Passage: data broadcast .................. 18941.984 ( 1.7018 %)
Message Passage: data reduction .................. 266.122 ( 0.0239 %)
Message Passage: data gathering .................. 1585.258 ( 0.1424 %)
Message Passage: data scattering.................. 23399.430 ( 2.1022 %)
Message Passage: point data gathering ............ 22.878 ( 0.0021 %)
Total: 836276.845 75.1326

All percentages are with respect to total time = 1113067.298

Post Reply