Drew,
Yes, that completely makes sense.
I’ve been talking with the BlueWaters user support team and they mentioned that MPI does not mix well with multiprocessing.pool on Cray systems such as BlueWaters. “MPI stacks often are no very tolerant of calling fork() (which Pool does) since the fork()ed childing all try to access the same network hardware (at a low level since MPI wants to be very fast).” We also discussed whether this is a common issue with supercomputers, and they said that " It is much more likely to work on a “whitebox cluster” such as eg a campus compute cluster or most XSEDE machines (which are partially designed with this is mind) than on the heavy duty, purely HPC-only, Cray machines that are Blue Waters and the large DOE machines."
At first, we discussed the various fixes that could be employed, and a few more complicated fixes such as loading a newly compiled MPI created by them on BlueWaters for this problem. But, we realized that multiprocessing.pool is only utilized in pool.py. We checked if its use is performance-critical when running depletion on BlueWaters by altering the source code slightly, to not use pool.starmap, but instead use itertools.starmap. And we found that removing multiprocessing.pool did not have a huge impact on runtime. For example, I ran a simulation with 10 active, 1 inactive, 10000 particles, on 32 nodes and 32 openmp threads. The total runtime for one depletion step was 318 seconds, and the time spent in the starmap function for rank0 was only 1.6 seconds (which is ~0.5% of runtime).
I’m sure that the use of multiprocessing.pool speeds up the depletion run slightly, but it seems to not be completely necessary especially since it is only used with rank0. I’m not sure if anyone else using CRAY machines has ever come across this issue, or perhaps it is only a BlueWaters issue since it is a pretty old machine. With that, I’ll probably run my simulations with this slight change in the source code implemented, just so that it works on BlueWaters.
Thanks for your help,
Gwen