Multi-node job do not scale

User Support
1

Greetings,

Firstly, i wish a happy new year to everyone in the OpenMC community.

I am having difficulties to run a multi-node job in a cluster. I am able to run my model without any errors using one computing node. However, if i increase the number of requested nodes, the speed and the efficiency (number of particles/second) of the simulation do not change accordingly. I would really appreciate any help to diagnose the problem.

Some specifications of the system i am using are shown in the table below:

Specification Value
Model Intel Xeon CPU E5-2670 v3 @ 2.30GHz
Cores per node 24 cores on two sockets (12 cores/socket)
Threads per core 2
Threads per node 24 x 2 = 48
Clock rate 2.3 GHz nominal
RAM 64 GB DDR4
Local storage 0 GB - diskless
OS Suse Linux Enterprise Server (SLES)

It is also important to note that there are two NUMA nodes per computing node (i.e. 12 physical cores per NUMA node).

The script i use to schedule the job is:

#!/bin/bash

#PBS -N BEAVRS_HZP
#PBS -l select=2:ncpus=48:mpiprocs=2
#PBS -l walltime=01:00:00
#PBS -r n
#PBS -o /scratch/60032a/ssartur/openmc_output_files
#PBS -e /scratch/60032a/ssartur/openmc_output_files
#PBS -m abe
#PBS -M ssartur@poli.ufrj.br

cd /scratch/60032a/ssartur/openmc_input_files

source /sw/apps/python/miniconda3/4.12.0-py37/bin/activate montecarlo-openmpi

python run_openmc.py

And the script i use to run OpenMC is:

import openmc

openmc.run(threads = 24, mpi_args = ['mpiexec', '-n', '4', '--bind-to', 'numa', '--map-by', 'numa'])

I would also appreciate help to clarify a few things:

  1. In the openmc.run command, does the mpi_args = ['mpiexec', '-n', 'N', '--bind-to', 'numa', '--map-by', 'numa'] argument sets the job’s total number of MPI processes or the number of MPI processes per node?

  2. In the openmc.run command, does the threads = N argument sets the job’s total number of threads, the number of threads per node or the number of threads per MPI process?

  3. If i have installed OpenMC with OpenMPI through Conda using the command conda install "openmc=*=*openmpi*", i don’t have to import the OpenMPI previously installed in the system using, for example, module load openmpi-gnu/4.1.1. Is that correct?

Thanks in advance,
Artur Souza

2

Hello, did you solve this problem, I also encountered this problem but I didn’t find a solution.