DagMC Geometry - "Open MC aborted unexpectedly"

User Support
1

Hey guys,

I have made a model of a reactor in CAD, and my project is to explore the dagmc/cubit toolchain in order to simulate the neutronics on the cad geometry in OpenMC.

To start, I am only trying to import one fuel volume. I’ve separated this volume from the rest of the model, exported it as a .sat file, and followed the openmc-specific steps (found here: Code-Specific steps for OpenMC — DAGMC) and then the pre-processing steps for cubit (found here: Cubit basics — DAGMC). The result is my fuel volume surrounded by a cuboid of finite thickness. First I save the model as a .h5m file, then using the command below, I exported this file as dagmc.h5m:

export dagmc dagmc.h5m

This starts a faceting process which takes a few minutes, then I move this file into the same directoy as my .py file for the openmc simulation. Using the code below, I only want to check that the geometry was created correctly:

import matplotlib
import openmc

u235 = openmc.Material(name="fuel")
u235.add_nuclide('U235', 1.0, 'ao')
u235.set_density('g/cc', 11)

mats = openmc.Materials([u235])
mats.export_to_xml()

settings = openmc.Settings()
settings.dagmc = True
settings.batches = 10
settings.inactive = 2
settings.particles = 5000
settings.export_to_xml()

p = openmc.Plot()
p.width = (25.0, 25.0)
p.pixels = (400, 400)
p.color_by = 'material'
p.colors = {u235: 'yellow'}
openmc.plot_geometry()

Running this .py file gives the following output:

                   | The OpenMC Monte Carlo Code
         Copyright | 2011-2021 MIT and OpenMC contributors
           License | https://docs.openmc.org/en/latest/license.html
           Version | 0.12.1
          Git SHA1 | 36913589c4f43b7f843332181645241f0f10ae9e
         Date/Time | 2021-08-10 14:25:16
    OpenMP Threads | 6

 Reading settings XML file...
 Reading cross sections XML file...
 Reading materials XML file...
 Reading DAGMC geometry...
Using the DOUBLE-DOWN interface to Embree.
Loading file dagmc.h5m
Initializing the GeomQueryTool...
Using faceting tolerance: 0.001
Building acceleration data structures...
Traceback (most recent call last):
  File "salt.py", line 27, in <module>
    openmc.plot_geometry()
  File "/opt/conda/lib/python3.7/site-packages/openmc/executor.py", line 59, in plot_geometry
    _run([openmc_exec, '-p'], output, cwd)
  File "/opt/conda/lib/python3.7/site-packages/openmc/executor.py", line 38, in _run
    raise RuntimeError(error_msg)
RuntimeError: OpenMC aborted unexpectedly.

Looking at the documentation here (openmc.plot_geometry — OpenMC Documentation), it seems like a RuntimeError from plot_geometry() means “openmc executable returns a non-zero status”. Feedback/thoughts on my approach to dagmc and on what might be going wrong would be appreciated.

Thanks for your time,
Luke

2

Hi @LukeLabrie,

Thanks for trying out DAGMC geometry in OpenMC!

Any chance you could drop the .h5m model at DAGMC Model Drops so I can try your script out and do some digging?

Another thing that would provide some more information is running OpenMC from the terminal to ensure that any errors being produced there aren’t suppressed.

3

edit: Sorry, just saw the link, dropped you the model there with my name in the description.

Hey Patrick,

Happy to drop you the model, but it’s not letting me upload it here. Do you know a way around that?

So when I am running that python file as a python command in the bash shell and that was the output. Is that what you mean?

4

Thanks! I should be able to take a look today or tomorrow.

So when I am running that python file as a python command in the bash shell and that was the output. Is that what you mean?

The openmc.plot_geometry() call in the Python script is running

$ openmc -p

under the hood, which reads the XML input files you’ve generated and runs OpenMC in plotting mode. Running this inside the Python script may be suppressing some useful output about the error occurring here. I’d recommend running the command above explicitly to see what that output looks like.

5

Gotcha. Running in terminal produces the following output:

 Reading settings XML file...
 Reading cross sections XML file...
 Reading materials XML file...
 Reading DAGMC geometry...
Using the DOUBLE-DOWN interface to Embree.
Loading file dagmc.h5m
Initializing the GeomQueryTool...
Using faceting tolerance: 0.001
Building acceleration data structures...
Illegal instruction (core dumped)

I’ll poke around the documentation to see what this might mean

6

I see you’re running with the double-down interface to Embree for the ray tracing kernel in DAGMC. Out of curiosity, what type of machine are you running this on?

I’m wondering if disabling that option (for now) when building DAGMC might fix this issue as Embree contains some hardware-specific instructions that might be getting applied here when they shouldn’t. Would you mind trying that out?

7

Interesting, I’m actually running it in Jonathan Shimwell’s docker container (openmc_workshop/Dockerfile at main · ukaea/openmc_workshop · GitHub) for his OpenMC workshop. I did this to avoid having to install dagmc, dependencies, and nuclear data. I’m running the container on manjaro linux but I don’t think that matters while I’m in the container. To disable it would I just comment out the install in the dockerfile?

8

Ah, I see! Yeah we may be running into an issue with architecture compatibility here. The easiest path forward for building that Docker image without double-down in DAGMC is to comment this line

and rebuild the image.

9

Progress, I think. I commented out the double_down line, ran from the .py file and it looks like openmc was able to complete a few more steps before aborting, see output below:

 Reading settings XML file...
 Reading cross sections XML file...
 Reading materials XML file...
 Reading DAGMC geometry...
Set overlap thickness = 0
Set numerical precision = 0.001
Loading file dagmc.h5m
Initializing the GeomQueryTool...
Using faceting tolerance: 0.001
Building acceleration data structures...

Implicit Complement assumed to be Vacuum
 Preparing distributed cell instances...
 Reading plot XML file...

 =======================>     PLOTTING SUMMARY     <========================

Plot ID: 1
Plot file: plot_1.ppm
Universe depth: -1
Plot Type: Slice
Origin: 0.0 0.0 0.0
Width: 25.0 25.0
Coloring: Materials
Basis: XY
Pixels: 400 400

 Processing plot 1: plot_1.ppm...
Traceback (most recent call last):
  File "salt.py", line 27, in <module>
    openmc.plot_geometry()
  File "/opt/conda/lib/python3.7/site-packages/openmc/executor.py", line 59, in plot_geometry
    _run([openmc_exec, '-p'], output, cwd)
  File "/opt/conda/lib/python3.7/site-packages/openmc/executor.py", line 38, in _run
    raise RuntimeError(error_msg)
RuntimeError: OpenMC aborted unexpectedly.

Let me know your thoughts. It took a very long time to run, but the fuel geometry I’m giving it is not simple so that could be why. I’m going to run it from terminal now to see if I can get more detail.

10

Here is the output running from terminal, looks like there is a “segmentation fault”. Any ideas?

 Reading settings XML file...
 Reading cross sections XML file...
 Reading materials XML file...
 Reading DAGMC geometry...
Set overlap thickness = 0
Set numerical precision = 0.001
Loading file dagmc.h5m
Initializing the GeomQueryTool...
Using faceting tolerance: 0.001
Building acceleration data structures...
Implicit Complement assumed to be Vacuum
 Preparing distributed cell instances...
 Reading plot XML file...

 =======================>     PLOTTING SUMMARY     <========================

Plot ID: 1
Plot file: plot_1.ppm
Universe depth: -1
Plot Type: Slice
Origin: 0.0 0.0 0.0
Width: 25.0 25.0
Coloring: Materials
Basis: XY
Pixels: 400 400

 Processing plot 1: plot_1.ppm...
Segmentation fault (core dumped)
11

As an update, re-compiled openmc with debug options on to get more detail (15. Troubleshooting — OpenMC Documentation), more to follow.

12

Ok so I ran the geometry from the terminal with the debug options and the error was “Ray fire query failed”. See output below:

 =======================>     PLOTTING SUMMARY     <========================

Plot ID: 1
Plot file: plot_1.ppm
Universe depth: -1
Plot Type: Slice
Origin: 0.0 0.0 0.0
Width: 25.0 25.0
Coloring: Materials
Basis: XY
Pixels: 400 400

 Processing plot 1: plot_1.ppm...
MOAB ERROR: --------------------- Error Message ------------------------------------
MOAB ERROR: Ray fire query failed!
MOAB ERROR: point_in_volume() line 907 in GeomQueryTool.cpp
MOAB ERROR: contains() line 824 in cell.cpp
openmc: /MOAB/moab/src/OrientedBoxTreeTool.cpp:923: virtual moab::ErrorCode moab::RayIntersectSets::leaf(moab::EntityHandle): Assertion `lastSet' failed.
Aborted (core dumped)

Any idea what might be causing that?

13

Thanks @LukeLabrie for the extra digging and additional info. Definitely helpful in determining what was going on.

On the Dockefile

The version of MOAB being installed in that Docker file (5.2.1) had a thread-safety issue that was resolved in the latest release (5.3.0). So another small tweak to the Dockerfile will be needed. Regarding the repo, I talked to the owner of the UKAEA openmc_workshop repo and they said it’s going to be removed soon in favor of this one.

I went ahead and submitted a PR there to address this issue.

You can try out that Dockerfile if you like (you’ll still want to remove the -DDOUBLE_DOWN=ON line for your machine to avoid the illegal instruction issue w/ Embree).

On the Model

I pulled your Python file from the initial post along with the DAGMC model you dropped in box (thanks!). After a couple of tweaks, I was able to run particles and generate this plot of the model:

plot_1
plot_1800×800 1.58 KB

which is hopefully what you were going for. I did have to update the plot origin and extents a bit, so here the final Python file I was using for convenience:

import matplotlib
import openmc

u235 = openmc.Material(name="fuel")
u235.add_nuclide('U235', 1.0, 'ao')
u235.set_density('g/cc', 11)

mats = openmc.Materials([u235])
mats.export_to_xml()

settings = openmc.Settings()
settings.dagmc = True
settings.batches = 10
settings.inactive = 2
settings.particles = 5000
settings.export_to_xml()

plots = openmc.Plots()

p1 = openmc.Plot()
p1.origin = (0.0, 0.0, 0.5)
p1.width = (500, 500)
p1.pixels = (800, 800)
p1.color_by = 'material'
p1.colors = {u235: 'yellow'}
plots.append(p1)
plots.export_to_xml()

openmc.plot_geometry()

Before I did this, I ran the model through the make_watertight program to seal any small gaps between surfaces that inevitably occur when tessellating the CAD geometry. (You can also apply this sealing algorithm during the initial export from Cubit with export dagmc "dagmc.h5m" make_watertight).

Ok, that was a lot. Let me know if you have any questions! Thanks again for sticking w/ this!

14

Success (with slight tweak described below)! Thanks for your help.

I re-exported the dagmc.h5m file using make_watertight, built a new docker image from Shimwell’s updated dockerfile, tweaked the .py file as above. On the first go, OpenMC was not recognising the settings.dagmc variable. That is, despite the line in the .py file

settings.dagmc = True

openmc was still looking for a geometry.xml file. Even if I commented out the

settings.export_to_xml()

command and just manually edited the .xml file generated on the last run to include

<dagmc>true</dagmc>

it would still search for the geometry.xml file. I noticed that Shimwell’s updated dockerfile pulls openmc from the dev branch, and I resolved this issue by reverting back to 0.12.1. So I’m not sure if this is an issue with the dev branch or the specific docker build, but I thought it worth mentioning.

Thanks again,
Luke

15

Also, out of curiosity, do you remember roughly how long it took you to run the model? It takes ~ 3 hours for me, wondering if that’s normal.

16

Hi Luke and Patrick

Sorry I didn’t noticed this earlier. Also sorry for moving the workshop I hope to put up a redirect notice soon.

Just wanted to mention the method of using dagmc is different in the development version of openmc

dag_univ = openmc.DAGMCUniverse('dagmc.h5m')
geom = openmc.Geometry(root=dag_univ)

Interesting to see Manjaro getting used, I didn’t know about the Double Down incompatibility there. I can add a flag to the Neutronics workshop dockerfile that allows it to be build without double down.

17

Just merged in an optional build parameter for the Dockerfile that might help with Double Down free building . The Dockerfile has been updated

1 Like
18

Good to know on the updated DagMC usage.

Thanks for the update. I saw your note in the Dockerfile – does DoubleDown significantly speed up the simulations? Wondering if I should try Ubuntu instead.

19

The percent of speed up will vary from model to model. But for my use cases, yes I see a significant speed up in both the transport time per particle and the initiation time of the simulation is quicker as well. I really appreciate the reduced initiation time in particular for for large models.

1 Like
20

Ah, yeah I should’ve clarified that newer versions of OpenMC do still require a geometry.xml file as we now allow DAGMC models to be inserted into CSG models as universes that fill a CSG cell. Glad you got it sorted out!

To expand on @Shimwell’s comments about Embree – yes, it should be faster than the default DAGMC ray tracer (the one that is part of MOAB) in both serial and multi-thread runs. The biggest effect I’ve noticed is that It will also scale better with the number of threads (MOAB wasn’t designed with threaded models in mind). Embree uses some hardware-specific instructions to accelerate ray queries in its data structures. It will also adjust the data structure design based on the available architecture. Depending on how Embree is configured, some of these instructions are Intel-specific, which is possibly why you were running into the Illegal Instruction error before.

If you’d prefer to have a stack built with Embree enabled, I’d be happy to help with that. I’ve read several posts on running Embree on AMD and such, so I think that’s possible if you’d like to give it a try.