Discrepancy between total simulation wallclock time, time spent in transport, and depletion time

User Support
1

I apologize in advance for the long post. I’m trying to provide as much relevant information as possible.

Model description:

I am modeling a simple MSR, and have two models, A and B. Both models have the same geometry, consisting of concentric spheres/spherical shells of inner moderator, fuel, outer moderator, and a blanket.

I am performing transport-coupled depletion simulations for both models, and the only difference between Model A and Model B is the number of material transfer rates.

  • Model A has a single transfer rate from the fuel to an off-gas system material representing the removal of Xe and Kr. Total number of depletable materials: 4
  • Model B has additional transfer rates between the internal (represented in the geometry) fuel, blanket, and moderator and their external (not modeled explicitly) counter parts. In this case the Xe and Kr are removed from the external fuel. Total number of depletable materials: 7

Simulation/run settings:

For both models I simulated 150,000 particles, 200 batches, 20 of which were inactive.

The calculations were performed on a HPC which uses slurm as a workload manager. I installed OpenMC in a conda environment, and when I run openmc -v while in the environment I obtain the following output:

OpenMC version 0.15.3

Commit hash: 27e38e894697bb32a1dac7848d2618818b6b8daf

Copyright (c) 2011-2025 MIT, UChicago Argonne LLC, and contributors

MIT/X license at < License Agreement — OpenMC Documentation >

Build type: Release

Compiler ID: GNU 14.3.0

MPI enabled: yes

Parallel HDF5 enabled: yes

PNG support: yes

DAGMC support: yes

libMesh support: no

MCPL support: no

Coverage testing: no

Profiling flags: no

UWUW support: n

For both models, the simulation is run using both OpenMP and MPI parallelism on a single node, with 8 MPI processes (tasks) and 16 OpenMP threads per MPI process.

Both models are depleted for a single 30 day step using the following integrator:

integrator = openmc.deplete.PredictorIntegrator(operator = operator, timesteps = [30], timestep_units = ‘d’, power = 1e6, solver = ‘cram16’)

The issue:

My question pertains to the run time and CPU utilization of these calculations.

  • Model A uses 4.7 hours (wallclock time), has a CPU utilization of 91.7%, and memory usage of 48.4 GiB.
  • Model B uses 7.8 hours (wallclock time), has a CPU utilization of 50.1%, and memory usage of 48.4 GiB.

The increased runtime for Model B initially made sense to me since the model has additional materials to deplete, but the significantly lower CPU utilization surprised me. After some reading on other posts ( Depletion simulation on supercomputer gets stuck between depletion steps. - #21 by pdeng and Determing real simulation time - #6 by Jarret ) I assumed that either my runs were getting stuck on the “creating openmc_simulation_nX.h5…” step or that I was simply observing a consequence of depletion being performed sequentially.

To investigate this further I looked at the runtime information contained in the depletion_results.h5 and openmc_simulation.h5 files. Here’s what I found + some quantities I calculated:

Model A:

  • Transport time: 16173.206191880001 seconds
  • Statepoint time: 0.06152004899999999 seconds
  • Burn-up time (average): 0.6852113008499146 seconds
  • Burn-up time * # depletable materials: 2.740845203399658 seconds
  • Total (slurm) time: 16920.0 seconds
  • transport + BU time (all mat): 16175.9470370834 seconds
  • CPU utilization (%): 91.7

Model B:

  • Transport time: 13319.936292557 seconds
  • Statepoint time: 0.058729711999999996 seconds
  • Burn-up time (average): 1677.14281725032 seconds
  • Burn-up time * # depletable materials: 11739.99972075224 seconds
  • transport + BU time (all mat) 25059.93601330924 seconds
  • Total (slurm) time: 28080.0 seconds
  • CPU utilization (%): 50.1

According to these files the program is spending barely anytime writing statepoint files for either model. When I sum the transport time and what I assume is the time performing transmutation calculations there is still a significant amount of time leftover in the total slurm wall-clock time. Almost 800 seconds for Model A and 3000 seconds for Model B. I took another look at the time information in the final statepoint.h5 file for both models and got the following:

B C
Key Model A statepoint.h5 Model B statepoint.h5
SEND-RECV source sites 0.031202494999999983 0.031676670000000004
accumulating tallies 0.5846612360000003 0.36081441200000003
active batches 8160.979186167 7433.907243442
inactive batches 367.416181962 355.833702045
reading cross sections 0.0 0.0
sampling source sites 0.160469112 0.16418679499999994
simulation 8528.395258101 7789.740831161
synchronizing fission bank 564.1339287830001 2017.9341377629996
total 8528.949728731 7790.286593861
total initialization 0.0 0.0
transport 7963.5831385599995 5771.347751955998
writing statepoints 0.005471432 0.0054026

From these numbers the “synchronizing fission bank” numbers stand out to me, as they are ‘relatively’ close to the above discrepancy. Then again, I may be imagining a connection where there is none.

I don’t know at what point openmc performs the fission bank synchronization, does this occur before the next transport calculation? Would this overlap with the program printing “creating state point …” line? I would guess that the time spent synchronizing fission banks depends on the number of MPI processes and the system architecture on which I’m running the calculation, but is there some way in which this time can be reduced? Or is it simply something I should approach using trial and error to see what job parameters give me the best performance? I’m also a little confused why Model B would have a fission bank synchronization time so much longer than Model A when the actual geometry in which neutrons are tracked is identical for both.

2

Hi Baltasar,

I would highly suggest setting up a script to monitor cpu usage and memory usage during the run and export the results to a spreadsheet to evaluate. The Cpu usage varies significantly within a run as you can see from the post you linked from me. Transport requires the use of the CPU, but depletion/burnup uses very little cpu and mostly memory. From your numbers, you see that your burn-up time is very low for model A, but significant in model B. I found from tests on my pc that during transport i used ~90 cpu% and during depletion I used ~5 cpu%, which if you multiply the usage percentage by simulation time of their respective step and then divide by the overall usage time as seen here,

(cpu% of transport * time of tranport + cpu% of depletion*time of depletion)/(time of transport+ time of depletion)

you get the ~91% and ~50% for run A and B respectively. So the real question is why your first test completed depletion so fast while your second did not. You said you have 4 depletable materials in problem A and 7 in problem B, can I ask why the discrepancy if all you are changing between both problems are the transfer rates? This could easily be the reason for what you are seeing. Problems A depletion is extremely fast making me question if it even underwent depletion. Lastly, I would suggest if you are evaluating computer computation for runs always do a small predictor run (like insignificantly small) since the first run on fresh fuel has most of the nuclide population matrices (besides those explicitly created in the materials definition) initially empty until the first run in which it populates each section according to whatever nuclear database you are using. Therefore, the first run in a depletion chain is signficantly faster than the others, and this can cause the actual computational times to vary as it will significantly underestimate the first transport steps computational time (which for the predictor is very significant).

3

Hi Jarret,

Thank you for your reply! Model A and Model B are identical in terms of geometry, i.e. they have the same four layers (inner mod, fuel, outer mod, blanket). The inner moderator and outer moderator are defined using the same material, so these four layers contain three depletable materials.

In Model A the only material transfer rate is:

fuel → OGS

where the OGS is not modeled in the geometry but is defined as a depletable material.

In Model B meanwhile there are three additional materials, fuel-external, moderator-external, and blanket-external. Model B then contains three additional depletable materials and transfer rates include:

fuel (geometry) → fuel-external

fuel-external → fuel (geometry)

and corresponding rates for the moderator and blanket.

Based on this I guess the burn-up matrix for Model B is a lot larger/more complicated than for Model A, but I don’t know if this difference is enough to account for the difference in burn-up time.

When you say “small predictor run” are you referring to the number of particles/generators? I will take another look at setting up a monitoring script and for now accept that the run-time is not unreasonable.

4

Hi Baltasar,

I would still expect longer burnup times then you got for Model A. I doubt depletion is even occurring, I would check the flux within this region and the material composition after depletion to see if there was actually a change. Less than a second for 30 days burnup is crazy low no matter the model. Especially when transport time was 16000 seconds. If depletion is not being completed like I expect it could be do to your source definition, a wrong boundary condition preventing neutrons to your depletable materials, or your geometry just has too much shielding between the source and the depletable materials.

As for what I said about the predictor run, I was referring to the predictor integrator within the depletion module. When you run the depletion module (any integrator) the first transport steps matrix only contains the materials you explicitly defined in the material.xml. The first depletion step will populate the entire matrix in relation to whatever depletion cross section library you are using. So depending on if you are using the full chain (1000s) or simplified Casl (~300) your matrix for the next transport run will be that size instead of the number of nuclides in your material.xml. This will significantly increase the transport time, usually to multiples higher time. This is not a concern if you are only using depletion module and not doing a series of them or trying to test how optimized they are. But if you are trying to determine which integrator to use for a series of depletions, not doing a depletion run beforehand will underestimate the first transports impact on computational efficiency, and depending on the integrator have a more significant impact on their overall efficiency if that makes sense to you.