Hi OpenMC family!
I’m doing depletion simulation in OpenMC and this is taking a while, so that’s motivation me to try run OpenMC in more that one computer to increase the speed. In the laboratory of my university have 4 computers with ArchLinux available to simulate at the same time like a cluster. I tried harder, but didn’t worked.
Step-by-step of what I did
First I read the OpenMC documentation, but there just say we need to supply the mpi_args argument to openmc.run(), like:
openmc.run(mpi_args=['mpiexec', '-n', '32'])
So I tried find information in internet and there say things like we set the SSH without password. So I created a openmc user for every computer [nuclear212, nuclear213, nuclear214, nuclear215] doing:
sudo useradd -m openmc
sudo usermod -aG wheel openmc
sudo passwd openmc
And in the main computer (nuclear215) I did:
ssh-keygen -t rsa
ssh-copy-id openmc@nuclear214
ssh-copy-id openmc@nuclear213
ssh-copy-id openmc@nuclear212
To install the OpenMC in every computer I did:
sudo pacman -Syu openmpi #to update the computer and install OpenMPI
paru -S miniconda3 #To install miniconda
echo "#MiniConda3" >> ~/.bashrc #We need to source miniconda on bashrc
echo "[ -f /opt/miniconda3/etc/profile.d/conda.sh ] && source /opt/miniconda3/etc/profile.d/conda.sh" >> ~/.bashrc
echo "" >> ~/.bashrc
conda config --add channels conda-forge #Here folowing the quick install guide
conda create -n openmc-env
conda activate openmc-env
conda install mamba
mamba install vtk openmc[version=0.14.0,build=dagmc_mpi_openmpi_py312h87dd1e5_101] #Chossing the build to make sure we take the OpenMC+OpenMPI
paru -S nuclear-data #to install cross-sections
echo "#OpenMC" >> ~/.bashrc #Put the cross-sections on bashrc
echo "var=`echo /opt/nuclear-data/*hdf5 | head -n1`" >> ~/.bashrc
echo "export OPENMC_CROSS_SECTIONS=$var/cross_sections.xml" >> ~/.bashrc
echo "" >> ~/.bashrc
Ok, so now we have OpenMC with OpenMPI in every computer and the main computer have access to the other’s with SSH without password. So I tried to run the commands specifing the hosts. As we are using OpenMPI with OpenMP we need just 1 node MPI per machine, because the OpenMP take care of use every threds avaliable of that machine. So I tried to run with command (just 2 computers as a first try):
openmc.run(mpi_args=['mpiexec', '-np', '2', '--hosts', 'nuclear215,nuclear214' ])
Give a error because MPI can’t find the openmc command. I figure out when I run ssh nuclear214 openmc the bash can’t find openmc command. But if I run ssh nuclear214 and after login run openmc, so bash find it. I read that because the non-interactive SSH do not run the bashrc. So I cheated… I put a symbolic link in /bin to the binary openmc from the conda environment:
sudo ln -s /home/openmc/.conda/envs/openmc-env/bin/openmc /bin/
When I run, the MPI complain can’t find orted, so I did the same with orted:
sudo ln -s /home/openmc/.conda/envs/openmc-env/bin/orted /bin/
Ok, now I run and IT WORKS?!?!
So so…:
The OpenMC instance in main computer (nuclear215) it’s running, loading nuclides, etc. But the secondary instance (nuclear214) send a message “can’t find the XML input files”, so crash the MPI.
Questions
1 - Isn’t the Python API responsible for sending the XML files that it generalizes to all machines that OpenMC will run on?
2 - How MPI can fint the openmc binary in other machines using the conda envrolvment in right way?
3 - Am I on the right or wrong path to being able to run OpenMC in cluster mode?
4 - How MPI works to speedup the simulation of transport? I mean, it share particles between the machines? Or both machines simulate separate and take a average of the result?
Thanks guys, I’m new in the MPI world, first time trying it!
I’m a master degree studant of Nuclear Engennier at UFMG, a brasilian university