Performance issues using floats

[lcbernardo]

Dear ROMS users,

I have encountered a strange situation when I use floats in ROMS (or in my case, COAWST using only ROMS). My application's grid size is 300 x 300, with 15 vertical layers. And I am trying to run a 60-day simulation with floats, and the total number of floats released is 149975. I started off using 28 cpus (4 x 7 tiling) and I got some initial results. I then hoped to decrease the run time by using more nodes in our computing cluster, and tried to run using 56 cpus (7 x 8 tiling) and then 112 cpus (8 x 14 tiling). However, what occurred was the complete opposite, and the number of days simulated successfully decreased the more cpus I used. Has anyone encountered such a situation before?

It may have to do with our system's settings, so I'm still checking. But nevertheless, I'd appreciate any help on this issue.

Thanks,
Lawrence

[jcwarner]

as you increase the number of procs for the job, then the tile size decreases. 300x300 tiled 8x14 gives a tile 38 x 20 cells. that is not very big. the job is spending more time exchanging information than performing computations.
here are some previous posts in this issue
viewtopic.php?f=29&t=134

[lcbernardo]

Thank you for the reply Dr. Warner. I'll look into these previous posts to see what may be helpful. But just basing on the explanation you provided, if the problem is the added time exchanging information among tiles, will this be particularly true for the floats option? Because if I don't use floats, and also for the case wherein I couple ROMS and SWAN, I get increased performance when I increase the number of cpus from 28 to 56 or 112.

[kate]

The floats implementation works for a modest number of floats, but doesn't scale well. Yes, the floats incur a lot of communications and processing as they figure out which tile each float is on every timestep.

[lcbernardo]

Thanks for the reply Kate! So scaling isn't so good for floats. I guess I'll have to be more strategic in my use of this option then.