General Notes

• Jobs are run on the cluster using: qsub pbs-script
• You can check on PBS directives using the man pages (man qsub will give you a list).
• The commands following the #PBS lines are then executed just as they would if you had run them interactively.
• PBS scripts start from the user's $HOME directory; the environment variable $PBS_O_WORKDIR holds the name of the directory from which the job was submitted.
• The environment variable $PBS_NODEFILE contains the list of compute nodes assigned to the job by the PBS scheduler.

Example: MPI over Ethernet using mpiexec

• Check the current versions of compilers and MPICH: module avail
• This sample script requests 8 compute nodes (16 processors) for 15 minutes.
• The debug queue is specified.

#!/bin/bash
#PBS -l walltime=00:15:00
#PBS -l nodes=8:ppn=2
#PBS -M user@domain
#PBS -m e
#PBS -N cpi
#PBS -o pbs-cpi-pgi.out
#PBS -j oe
#PBS -q debug
#
#
# Executable Name (customize!)
#
cd $PBS_O_WORKDIR
echo "working directory = "$PBS_O_WORKDIR
. $MODULESHOME/init/sh
module load mpich/pgi-7.0/ch_p4/1.2.7p1
module list
ulimit -s unlimited
ulimit -c 0
ulimit -a
#
# Run Job
which mpiexec
mpiexec -v ./cpi-pgi-7-p4
#
echo "All Done!"

Description of PBS Directives

Comments to the shell, but significant to PBS (#PBS):

• -m e
  • Send email to the user when the job ends. You can also send mail when the job begins or aborts.
• -l nodes=8:ppn=2
  • Resource request, in this case request of eight compute nodes, with 2-processors on each.
• -l walltime=00:15:00
  • Resource request, for wallclock time of 15 minutes.
• -M user@domain
  • email address to use for notification.
• -j oe,
  • joins the standard output and error streams (otherwise you get them in separate files).
• -o pbs-cpi-pgi.out
  • renames the output stream (thanks to the above line, it will also include the standard error).
• -N cpi
  • gives the job a name - not really necessary (the name will show up under a qstat query).

Example: MPI over Myrinet using mpirun

• Check the current versions of compilers and MPICH: module avail
• This sample scripts requests 4 compute nodes (8 processors) with Myrinet (GM) for 1 hour.
• The property tag 'GM in the line "#PBS -l nodes=4:GM:ppn=2" requests compute nodes that have a Myrinet interface.
  • Specifying this property will ensure that a job is only allocated nodes from the Myrinet portion of the cluster. Myrinet offers much higher bandwidth (~240MB/s) and lower latency (~7 microseconds) then gigabit ethernet.
• The pbsnodes command will show the properties of any compute node.

#!/bin/bash
#PBS -l walltime=01:00:00
#PBS -l nodes=4:GM:ppn=2
#PBS -M user@domain
#PBS -m e
#PBS -N test
#PBS -o pbs-cpi-intel.out
#PBS -j oe
#
#
cd $PBS_O_WORKDIR
echo "working directory = "$PBS_O_WORKDIR
export NN=`cat $PBS_NODEFILE | wc -l`
echo "NN = "$NN
. $MODULESHOME/init/bash
module load mpich/intel-10.0/ch_mx/1.2.7..4
module list
ulimit -s unlimited
ulimit -c 0
#
# Run Job
which mpirun
mpirun -v -machinefile $PBS_NODEFILE -np $NN ./cpi-intel-10-mx
echo "All Done!"

More Sample Scripts

• The directory /util/pbs-scripts on u2.ccr.buffalo.edu contains sample script for running many different applications.