Hybrid MPI with OMP and MPICH

User Support
1

Hi, I’m trying to use parallel calculation of OpenMC.

I’m wondering if it is possible to use both OpenMP and MPICH2.

For example, use MPICH2 for communicating between nodes and use OpenMP for parallel calculating in a node.

Thanks,
Kyungkwan.

2

Hi Kyungkwan,
As of OpenMC version 0.5.3 (released a few days ago), you may utilize OpenMP in addition to MPI distributed parallelism.
Adam

3

Thank you for reply.

So, to use OpenMP with MPI, do I have to compile with MPICH2?

Also, to get the latest version in Linux, all I have to do is get the source via Git?

Thanks, Kyungkwan

2013년 11월 27일 수요일 오후 7시 51분 43초 UTC+9, Adam Nelson 님의 말:

4

Yes, you will have to compile with MPICH2 but also compile with OpenMP (use the flag OPENMP=yes when executing the make command). You are correct, the latest version can be obtained with git (or directly here: https://github.com/mit-crpg/openmc/archive/v0.5.3.zip).

I hope this helps,
Adam

5

Thanks, that helped a lot.

What about the command line input, when using hostfile (or machinefile) to indicate which nodes are to be used?

For example, I’m currently using mpirun -hostfile [hostfile name] openmc

Thanks, Kyungkwan

2013년 11월 28일 목요일 오전 3시 7분 12초 UTC+9, Adam Nelson 님의 말:

6

Hi Kyungkwan,

The format for the machinefile will depend on the particular MPI implementation you are using. For MPICH, you can consult the documentation here: http://www.mpich.org/documentation/guides/. Just to pick a snippet from their documetation:

    The 'machinefile' is of the form:

      host1
      host2:2
      host3:4   # Random comments
      host4:1

    'host1', 'host2', 'host3' and 'host4' are the hostnames of the
    machines you want to run the job on. The ':2', ':4', ':1' segments
    depict the number of processes you want to run on each node. If
    nothing is specified, ':1' is assumed.

Hope this helps!

Best regards,
Paul

7

Thanks a lot for your reply, Paul.

Still I have a question. I’m using OpenMPI now. When I compile the source, I set MPI=yes, OpenMP=yes.
Then, when I run OpenMC with hostfile, something funny happened.

For example, I have two nodes and each has 12 processor, so I wrote the hostfile as
node01 slots=12
node02 slots=12

When I run it, however, the load exceeded 12. It even reached 100 or higher.
What happened?

The older version that I used without OpenMP didn’t have this kind of problem.

Thanks.

2013년 11월 30일 토요일 오전 2시 11분 22초 UTC+9, Paul Romano 님의 말:

8

If you want to run with MPI across nodes and OpenMP within nodes, you should only be initiating two MPI processes where each MPI process will be running with 12 OpenMP threads. Thus, your hostfile probably shouldn’t have the slots=12 or else both MPI processes will start on the same node.