Hello,everyone!
Recently, I plan to build my own server in the office. And when I was selecting CPUs，I was torn between choosing AMD's Epyc or Intel's Xeon. EPYC 9754 has more cores and higher cost performance than Xeon(Dual EPYC 9754 has 256 cores and 512 threads and I think this already meets my needs). But I have never run the ROMS with AMD before, it seems like AMD's CPU has poorer compatibility compared to Intel's?? (Sorry, I have no understanding about it)
I have tried to find some information about it from forum, And Porf.Jamie posted some helpful benchmarking test about Epyc and Xeon: viewtopic.php?f=29&t=4665
But seven years have passed, is EPYC still a good choice for running ROMS? I hope to receive some suggestions, thank you!
Enzo
CPU selection for self built servers
Re: CPU selection for self built servers
Dear Enzo --
My first advice is to benchmark. Most server manufactures will allow you to bench mark systems remotely. How does your ROMS configuration scale with cores?
My second advice is not to chase cores or threads. All my experience suggests that ROMS becomes quickly limited by bandwidth to memory. Now, the new DDR5 memory has lower latency and higher speed -- but the cores are also faster. I _strongly_ suspect that when the number of cores is 2 to 4 times the number of memory channels, you will see no further scaling with cores. But benchmark!
I have never had any compatibility problems with ROMS and epyc, and have servers up to the Rome generation. The newer Epyc generations even have AVX512, thought I have not found that too helpful before. I have used Intel and AMD processors extensively, and have had no problem with either.
Again, in my experience, memory channels and memory bandwidth are more important than raw core count, once you have enough cores.
Jamie
My first advice is to benchmark. Most server manufactures will allow you to bench mark systems remotely. How does your ROMS configuration scale with cores?
My second advice is not to chase cores or threads. All my experience suggests that ROMS becomes quickly limited by bandwidth to memory. Now, the new DDR5 memory has lower latency and higher speed -- but the cores are also faster. I _strongly_ suspect that when the number of cores is 2 to 4 times the number of memory channels, you will see no further scaling with cores. But benchmark!
I have never had any compatibility problems with ROMS and epyc, and have servers up to the Rome generation. The newer Epyc generations even have AVX512, thought I have not found that too helpful before. I have used Intel and AMD processors extensively, and have had no problem with either.
Again, in my experience, memory channels and memory bandwidth are more important than raw core count, once you have enough cores.
Jamie
Re: CPU selection for self built servers
Dear Prof.Jamie
Thank you for your prompt reply and contribution to all ROMS learners, it's really helpful. And Professor, I have reviewed your postsviewtopic.php?f=29&t=4665, one of these results is that dual CPU performs better. So, I think a dual EPYC9654 is good choice(single EPYC 9654 with 12 memory channels, 4800Mhz memory, 96 cores and 192 threads, CPU at 2.4GHz).
But from my understanding the number of memory channels for dual CPU and single CPU should be the same? Or is my understanding wrong? I'm a bit confused.
The second question is the choice of memory size. Is it possible that the running speed of ROMS will be limited by the memory size? Is 512g of memory better than 256g?
And dear Prof.Jamie, I also want to inquire about how to configure tiling. I found some information from the posts:viewtopic.php?f=17&t=2001&p=7771#p7714
regards,
Enzo
Thank you for your prompt reply and contribution to all ROMS learners, it's really helpful. And Professor, I have reviewed your postsviewtopic.php?f=29&t=4665, one of these results is that dual CPU performs better. So, I think a dual EPYC9654 is good choice(single EPYC 9654 with 12 memory channels, 4800Mhz memory, 96 cores and 192 threads, CPU at 2.4GHz).
But from my understanding the number of memory channels for dual CPU and single CPU should be the same? Or is my understanding wrong? I'm a bit confused.
The second question is the choice of memory size. Is it possible that the running speed of ROMS will be limited by the memory size? Is 512g of memory better than 256g?
And dear Prof.Jamie, I also want to inquire about how to configure tiling. I found some information from the posts:viewtopic.php?f=17&t=2001&p=7771#p7714
But in your posts:viewtopic.php?p=17955&hilit=amd#p17955, you used a tiling of 4x32 for the grid size of 2048*256. You seem to have used different tiling methods. Can you share some experience about tiling? Thank you soooo much!In the present grid:
Lm (#points in I-direction) = 1398 = 2 * 3 * 233, and
Mm (#points in J-direction) = 726 = 2 * 3 * 11 * 11
As I understand it NtileI and NtileJ should preferably be made up of the prime factors of Lm and Mm, as this would generate equally sized tiles. Fortunately for me the overhead of having unequally sized tiles is not that big.
Based on the testing that I have done, the best choice is NtileI=68 and NtileJ=33. This choice is 8 sec. (7%) faster than 8*81 for 1000 modeled time-steps.
regards,
Enzo
Re: CPU selection for self built servers
I would still benchmark, but unless you need all those cores for other reasons, I would go with a higher clockspeed, lower core count chip.
A dual CPU machine will have twice as many memory channels.
The memory size, if it is large enough for your model to run, will not affect the run speed, as long as all memory channels are populated. (24 memory channels can cost a bunch to populate).
If you are running in parallel with openMP, you have a great deal of flexibility in how you parallelize. If you are using openMPI, the number of tiles must match the number of MPI jobs. Which will give you the best performance on a single machine must be determined by experimentation, and is a function of the memory topology.
Again, see if your local systems provider can give you access to a machine to benchmark. I have NOT benchmarked these newer Epyc chips, my latest is the EPYC 7343. But my sense is 96 cores will be more than you can use efficiently for ROMS. For other projects, they can be very performant.
Jamie
A dual CPU machine will have twice as many memory channels.
The memory size, if it is large enough for your model to run, will not affect the run speed, as long as all memory channels are populated. (24 memory channels can cost a bunch to populate).
If you are running in parallel with openMP, you have a great deal of flexibility in how you parallelize. If you are using openMPI, the number of tiles must match the number of MPI jobs. Which will give you the best performance on a single machine must be determined by experimentation, and is a function of the memory topology.
Again, see if your local systems provider can give you access to a machine to benchmark. I have NOT benchmarked these newer Epyc chips, my latest is the EPYC 7343. But my sense is 96 cores will be more than you can use efficiently for ROMS. For other projects, they can be very performant.
Jamie
Re: CPU selection for self built servers
thank you, professor, wish you a happy happy life work! 