MPI
Message Passing Interface (MPI) is a communication protocol for parallel programming. MPI is specifically used to allow applications to run in parallel across a number of separate computers connected by a network.
Basic Features of MPI
Message passing programs generally run the same code on multiple processors, which then communicate with one another via library calls which fall into a few general categories:
- Calls to initialize, manage, and terminate communications.
- Calls to communicate between two individual processes (point-to-point).
- Calls to communicate among a group of processes (collective).
- Calls to create custom datatypes.
- Rich extended functionality, see some extended training materials here
Implementations of MPI
There are several different implementations of MPI available on the U2 cluster.
- Intel MPI (Recommended)
- This implementation has multi-network support (TCP/IP, Infiniband, Myrinet, etc.) - by default the best network is tried first.
- Compiler "wrappers" around both Intel's compiler suite (mpiifort, mpiicc, mpiicpc) and the GNU compilers (mpif90, mpicc, mpicxx)
- Show all current versions of Intel-MPI: module avail intel-mpi
- MVAPICH2
- This implementation runs over InfiniBand.
- Show all versions of MVAPICH2: module avail mvapich2
- MPICH 2 - A portable implementation of standard Message Passing Interface (MPI) created by the Argonne National Laboratory.
- OPENMPI - An open source implementation of MPI that is developed and maintained by a consortium made up of researchers from academia and industry.
- This implementation is network aware, so it will automatically select the network interface.
- OPENMPI is built specifically for a particular compiler.
- Show all the current versions of OPENMPI: module avail openmpi
Using MPI with GNU compilers
- Use the default Intel-MPI module to set paths to the compiler.
- See Basic Compilation for examples of compiling C and Fortran codes without MPI.
[user@edge mpi-stuff]$ module load intel-mpi/4.0
- Create a nodefile:
- In these examples the nodefile contains entries for the front-end machine (bono).
[user@edge mpi-stuff]$ cat nodefile edge edge edge edge [user@edge mpi-stuff]$
- In these examples the nodefile contains entries for the front-end machine (bono).
Compiling with MPI C
- Code: cpi from MPI test suite.
- Compilation:
[user@edge mpi-stuff]$ mpiicc -o cpi cpi.c
- Starting Daemons:
[user@edge mpi-stuff]$ mpdboot -f nodefile -v running mpdallexit on edge LAUNCHED mpd on edge via RUNNING: mpd on edge [user@edge mpi-stuff]$
- Running the code:
[user@edge mpi-stuff]$ mpiexec -n 2 ./cpi Process 0 of 2 on edge.ccr.buffalo.edu Process 1 of 2 on edge.ccr.buffalo.edu pi is approximately 3.1415926544231318, Error is 0.0000000008333387 wall clock time = 0.000336 [user@edge mpi-stuff]$
- Stopping Daemons:
[user@edge mpi-stuff]$ mpdallexit
Compiling with MPI Fortran
- Code: fpi from MPI test suite, modified to not run interactively.
- Compilation:
[user@edge mpi-stuff]$ mpif77 -o fpi fpi.f
- Starting Daemons:
[user@edge mpi-stuff]$ mpdboot -f nodefile -v running mpdallexit on edge LAUNCHED mpd on edge via RUNNING: mpd on edge [user@edge mpi-stuff]$
- Running the code:
[user@edge mpi-stuff]$ mpiexec -n 2 ./fpi Process 1 of 2 is alive Process 0 of 2 is alive pi is approximately: 3.1415926569231196 Error is: 0.0000000033333265 [user@edge mpi-stuff]$
- Stopping Daemons:
[user@edge mpi-stuff]$ mpdallexit
Tutorials on MPI
Running Programs on the Cluster
Sample PBS Scripts
Parallel Computing
Cluster Resources
The U2 cluster provides extensive resources for parallel computing.
- Schematic Design of the U2 Cluster:

- Networks:
- Gigabit Ethernet
- Myrinet
- Infiniband
- > 2112 processors
- > 45TB of global disk storage
Running Interactively
Using the Login Machine
- If you wish to run an MPI code interactively on the front-end machine (only for a few processes and short duration, please - otherwise use the batch system), you can simply launch using mpirun:
- mpirun -np 2 ./mine.x which in this case is for two processes.
Using a Compute Node
- Machines within the cluster are available for interactive use through the batch scheduler.
