Multi-node parallel in openmc.run() and integrator.integrate()

User Support
1

Hello. If i want to run a py file that contains ‘openmc.run()’ and ‘integrator.integrate()’.How could I make the simulation parellel? On a supercomputer, if I setting : ‘mpirun -map-by socket -bind-to socket python -u for_cal/klt_40_ARO.py’, the ‘integrator.integrate()’ can run paralleling.But when I try to use ‘openmc.run()’,it reports an error :

[cli_2]: write_line error; fd=5 buf=:cmd=init pmi_version=1 pmi_subversion=1
:
[cli_2]: write_line error; fd=5 buf=:cmd=init pmi_version=1 pmi_subversion=1
:
system msg for write_line failure : Bad file descriptor
[cli_2]: Unable to write to PMI_fd
[cli_2]: write_line error; fd=5 buf=:cmd=get_appnum
:
system msg for write_line failure : Bad file descriptor
Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(59)....: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(221): 
MPID_Init(72)........: 
init_local(135)......: channel initialization failed
init_pg(332).........: 
MPIR_pmi_init(110)...: PMI_Get_appnum returned -1
[cli_2]: write_line error; fd=5 buf=:cmd=abort exitcode=673828367
:
system msg for write_line failure : Bad file descriptor
[cli_3]: write_line error; fd=6 buf=:cmd=init pmi_version=1 pmi_subversion=1
:
system msg for write_line failure : Bad file descriptor
[cli_3]: Unable to write to PMI_fd
[cli_3]: write_line error; fd=6 buf=:cmd=get_appnum
:
system msg for write_line failure : Bad file descriptor
Fatal error in internal_Init: Other MPI error, error stack:
internal_Init(59)....: MPI_Init(argc=(nil), argv=(nil)) failed
MPII_Init_thread(221): 
MPID_Init(72)........: 
init_local(135)......: channel initialization failed
init_pg(332).........: 
MPIR_pmi_init(110)...: PMI_Get_appnum returned -1
[cli_3]: write_line error; fd=6 buf=:cmd=abort exitcode=673828367
:
system msg for write_line failure : Bad file descriptor
Traceback (most recent call last):
Traceback (most recent call last):
  File "for_cal/klt_40_ARO.py", line 986, in <module>
  File "for_cal/klt_40_ARO.py", line 986, in <module>
    openmc.run()
  File "/public3/home/scg8288/miniconda3/envs/openmc/lib/python3.8/site-packages/openmc/executor.py", line 314, in run
    openmc.run()
  File "/public3/home/scg8288/miniconda3/envs/openmc/lib/python3.8/site-packages/openmc/executor.py", line 314, in run
    _run(args, output, cwd)
  File "/public3/home/scg8288/miniconda3/envs/openmc/lib/python3.8/site-packages/openmc/executor.py", line 125, in _run
    _run(args, output, cwd)
  File "/public3/home/scg8288/miniconda3/envs/openmc/lib/python3.8/site-packages/openmc/executor.py", line 125, in _run
    raise RuntimeError(error_msg)
RuntimeError: OpenMC aborted unexpectedly.
    raise RuntimeError(error_msg)
RuntimeError: OpenMC aborted unexpectedly.

This error can be solved by changing ‘mpirun -map-by socket -bind-to socket python -u for_cal/klt_40_ARO.py’ to ‘python -u for_cal/klt_40_ARO.py’ and ‘openmc.run(mpi_args=[‘mpiexec’, ‘-n’, ‘10’])’,but ‘integrator.integrate()’ will lose it’s parellel.
I don’t see any multi-node parallel parameter settings in ‘integrator.integrate()’. At first, I want to use the Python API ‘crit_search’ to get a critical parameter to make the model be critical, then carrying out depletion calculation. But now it seems that I can only make one of these two steps multi-node parallel, if I want both of then parallel, how could I do? * I would feel very grateful if someone can help me!

2

openmc.run() and integrator.integrate() work differently from each other. For openmc.run, a subprocess is created to call the openmc executable directly, which is why mpi_args are passed in order to make the corresponding call. Depletion with integrator.integrate on the other hand uses the OpenMC shared library through the Python ctypes module, so it is directly a part of the original Python process, which is why the Python process itself needs to be invoked with mpiexec. At this time, there is no way to have a single script that does both of those. If you can, I would try breaking it up into multiple scripts, one that calls openmc.run and another that calls integrator.integrate.

3

Thank you very much ! I’ll try it.