Restarting a Simulation (wih openmc.lib.run)

User Support
1

I am trying to use the restart simulation feature to add more statistics to a previously run simulation without having to redo the whole thing.

However, I am having some difficulties.

I set up my simulation and run it using Python bindings to the C/C++ API:

openmc.lib.run(output=True)

Then I check if I have acquired sufficient statistics on a desired tally. If the relative uncertainty is too high, I then run:

openmc.lib.statepoint_load(f'statepoint.{no_batches}.h5')
no_batches = no_batches + 50  # no_batches is previously defined
openmc.lib.settings.set_batches(no_batches)  # Add 50 additional batches

if comm.rank == 0:
    print(f"new number of batches is {openmc.lib.settings.get_batches()}")
openmc.lib.run(output=True)

I use statepoint_load() as I understand that openmc.lib.run(restart_file=<file>) doesn’t exist (compared to openmc.run(restart_file=<file>) from the User Guide section 7.7).

The code above runs, however it seems to run the entire simulation again. Here is a part of the terminal output from running initially 50 batches and then adding 50 more because the uncertainty in a given tally was too high:

Uncertainty 0.30160626712065886 exceeds maximum threshold.
Restarting simulations and adding 50 batches
new number of batches is 100
Maximum neutron transport energy: 20000000 eV for N15

===============>     FIXED SOURCE TRANSPORT SIMULATION     <===============

Simulating batch 1
Simulating batch 2
Simulating batch 3
Simulating batch 4
Simulating batch 5
Simulating batch 6
Simulating batch 7
Simulating batch 8
Simulating batch 9
Simulating batch 10
Simulating batch 11
Simulating batch 12
Simulating batch 13
Simulating batch 14
Simulating batch 15
Simulating batch 16
Simulating batch 17
Simulating batch 18
Simulating batch 19
Simulating batch 20
Simulating batch 21
Simulating batch 22
Simulating batch 23
Simulating batch 24
Simulating batch 25
Simulating batch 26
Simulating batch 27
Simulating batch 28
Simulating batch 29
Simulating batch 30
Simulating batch 31
Simulating batch 32
Simulating batch 33
Simulating batch 34
Simulating batch 35
Simulating batch 36
Simulating batch 37
Simulating batch 38
Simulating batch 39
Simulating batch 40
Simulating batch 41
Simulating batch 42
Simulating batch 43
Simulating batch 44
Simulating batch 45
Simulating batch 46
Simulating batch 47
Simulating batch 48
Simulating batch 49
Simulating batch 50
Creating state point statepoint.050.h5…
Simulating batch 51
Simulating batch 52
Simulating batch 53
Simulating batch 54
Simulating batch 55
Simulating batch 56
Simulating batch 57
Simulating batch 58
Simulating batch 59
Simulating batch 60
Simulating batch 61
Simulating batch 62
Simulating batch 63
Simulating batch 64
Simulating batch 65
Simulating batch 66
Simulating batch 67
Simulating batch 68
Simulating batch 69
Simulating batch 70
Simulating batch 71
Simulating batch 72
Simulating batch 73
Simulating batch 74
Simulating batch 75
Simulating batch 76
Simulating batch 77
Simulating batch 78
Simulating batch 79
Simulating batch 80
Simulating batch 81
Simulating batch 82
Simulating batch 83
Simulating batch 84
Simulating batch 85
Simulating batch 86
Simulating batch 87
Simulating batch 88
Simulating batch 89
Simulating batch 90
Simulating batch 91
Simulating batch 92
Simulating batch 93
Simulating batch 94
Simulating batch 95
Simulating batch 96
Simulating batch 97
Simulating batch 98
Simulating batch 99
Simulating batch 100
Creating state point statepoint.100.h5…

From what I can see, it reruns the initial 50 batches, creates a second statepoint file (named statepoint.050.h5 since statepoint.50.h5 already exists from the initial simulation run), and then simulates the newly added 50 batches to reach a total of 100 batches.

So, perhaps in all simplicity, my question is: Is there a way to append batches or restart a simulation from the previus endpoint with openmc.lib?

(Note: I use openmc.lib for access to MPI communicators as I run the OpenMC simulations across multiple nodes on a cluster.)

1 Like
2

Commenting as I am also curious about this.

1 Like
3

So I figured it out.

The openmc.lib.init() can take command line arguments and pass them to the openmc command. Which means you can use the openmc -r <nameofstatepointfile>.h5 option described in 7.7 in the manual.

That means the resulting code could look something like this:

# Step 1
# Setup and run important simulation that would surely win the Nobel Prize in Physics and change your life forever.
...
openmc.lib.init(intracomm=comm)
openmc.lib.run(output=True)
comm.barrier()
openmc.lib.finalize()

# Step 2
# Check if statistics are good enough to acquire said Nobel Prize
boolean_type, rel_uncertainty = some_function_that_checks_std(f"statepoint.{no_batches}.5", uncertainty_goal)
            
# Step 3
# Restart simulation      
if boolean_type == True:
   print("Statistics are not sufficient for Nobel Prize")
   # I had to rename the summary.h5 file, but maybe you won't
   try:
      os.rename(f'summary.h5', f'summary.{no_batches}.h5')
   except:
     print("Already renamed the summary file")
   # Print statement when using master thread to avoid clutter in logfile
   if comm.rank == 0:
      print(f"Uncertainty {rel_uncertainty} exceeds maximum threshold.")
      print("Restarting simulations and adding more batches") 

   # The openmc -r statepointfile.h5 is what triggers the restart
   openmc.lib.init(args=["-r", f'statepoint.{no_batches}.h5'], intracomm=comm)
   
   # Here I just calculate the new number of batches assuming the uncertainty scales with 1/sqrt(no_particles)
   batch_factor = rel_uncertainty / uncertainty_goal
   new_no_batches = np.round(no_batches*batch_factor**2)
   if new_no_batches != no_batches:
      no_batches = int(new_no_batches)
   else:
      no_batches = int(no_batches + 100)

   openmc.lib.settings.set_batches(no_batches)
   if comm.rank == 0:
      print(f"new number of batches is {openmc.lib.settings.get_batches()}")
   openmc.lib.run(output=True)
   comm.barrier()
   openmc.lib.finalize()
else:
   print("Statistics was sufficient for Nobel Prize and no more batches were needed")

So I really just needed to re-initialize the simulations with openmc.lib.init(args=["-r", f'statepoint.{no_batches}.h5'], intracomm=comm).
This point could perhaps be added to section 7.7, if people deem it relevant.

1 Like