MPI

Message Passing Interface (MPI) is a communication protocol for parallel programming. MPI is specifically used to allow applications to run in parallel across a number of separate computers connected by a network.

Basic Features of MPI

Message passing programs generally run the same code on multiple processors, which then communicate with one another via library calls which fall into a few general categories:

• Calls to initialize, manage, and terminate communications.
• Calls to communicate between two individual processes (point-to-point).
• Calls to communicate among a group of processes (collective).
• Calls to create custom datatypes.
• Rich extended functionality, see some extended training materials here

Implementations of MPI

There are several different implementations of MPI available on the U2 cluster.

• Intel MPI (Recommended)
  • This implementation has multi-network support (TCP/IP, Infiniband, Myrinet, etc.) - by default the best network is tried first.
  • Compiler "wrappers" around both Intel's compiler suite (mpiifort, mpiicc, mpiicpc) and the GNU compilers (mpif90, mpicc, mpicxx)
  • Show all current versions of Intel-MPI: module avail intel-mpi
• MVAPICH2
  • This implementation runs over InfiniBand.
  • Show all versions of MVAPICH2: module avail mvapich2
• MPICH 2 - A portable implementation of standard Message Passing Interface (MPI) created by the Argonne National Laboratory.
  • MPICH is built specifically for a network interface and compiler. The U2 cluster has two internal networks, Gigabit ethernet and Myrinet. The compilers are GNU, Intel, and PGI.
  • NOTE: The MPICH 1 (MPICH) implementation is now deprecated on the CCR cluster. Instead, please use Intel MPI.
• OPENMPI - An open source implementation of MPI that is developed and maintained by a consortium made up of researchers from academia and industry.
  • This implementation is network aware, so it will automatically select the network interface.
  • OPENMPI is built specifically for a particular compiler.
  • Show all the current versions of OPENMPI: module avail openmpi

Using MPI with GNU compilers

• Use the default Intel-MPI module to set paths to the compiler. 
• See Basic Compilation for examples of compiling C and Fortran codes without MPI.

   [user@edge mpi-stuff]$ module load intel-mpi/4.0
  

• Create a nodefile:
  • In these examples the nodefile contains entries for the front-end machine (bono).

     [user@edge mpi-stuff]$ cat nodefile
     edge
     edge
     edge
     edge
     [user@edge mpi-stuff]$
    

Compiling with MPI C

• Code: cpi from MPI test suite.
  • view cpi code
• Compilation:

   [user@edge mpi-stuff]$ mpiicc -o cpi cpi.c
  

• Starting Daemons:

  [user@edge mpi-stuff]$ mpdboot -f nodefile -v
  running mpdallexit on edge
  LAUNCHED mpd on edge  via
  RUNNING: mpd on edge
  [user@edge mpi-stuff]$
  

• Running the code:

  [user@edge mpi-stuff]$ mpiexec -n 2 ./cpi
  Process 0 of 2 on edge.ccr.buffalo.edu
  Process 1 of 2 on edge.ccr.buffalo.edu
  pi is approximately 3.1415926544231318, Error is 0.0000000008333387
  wall clock time = 0.000336
  [user@edge mpi-stuff]$
  

• Stopping Daemons:

  [user@edge mpi-stuff]$ mpdallexit
  

Compiling with MPI Fortran

• Code: fpi from MPI test suite, modified to not run interactively.
  • view fpi code
• Compilation:

   [user@edge mpi-stuff]$ mpif77 -o fpi fpi.f
  

• Starting Daemons:

  [user@edge mpi-stuff]$ mpdboot -f nodefile -v
  running mpdallexit on edge
  LAUNCHED mpd on edge  via
  RUNNING: mpd on edge
  [user@edge mpi-stuff]$
  

• Running the code:

  [user@edge mpi-stuff]$ mpiexec -n 2 ./fpi
   Process  1 of  2 is alive
   Process  0 of  2 is alive
   pi is approximately: 3.1415926569231196  Error is: 0.0000000033333265
  [user@edge mpi-stuff]$
  

• Stopping Daemons:

  [user@edge mpi-stuff]$ mpdallexit
  

Tutorials on MPI

Running Programs on the Cluster

Sample PBS Scripts

Parallel Computing

Cluster Resources

The U2 cluster provides extensive resources for parallel computing.

• Schematic Design of the U2 Cluster:
• Networks:
  • Gigabit Ethernet
  • Myrinet
  • Infiniband
• > 2112 processors
• > 45TB of global disk storage

Running Interactively

Using the Login Machine

• If you wish to run an MPI code interactively on the front-end machine (only for a few processes and short duration, please - otherwise use the batch system), you can simply launch using mpirun:
  • mpirun -np 2 ./mine.x which in this case is for two processes.

Using a Compute Node

• Machines within the cluster are available for interactive use through the batch scheduler.

Tutorials on Parallel Computing