Using Infiniband on nyx

There are six nodes on the cluster nyx.engin.umich.edu that have Infiniband networking available for use by users of nyx. Infiniband is a low latency high bandwidth network with speeds up to 10Gbps and latency of less than 10 usec. Infiniband allows codes that rely heavily on communication to scale to more cpus with higher efficiency.

To use Infiniband you must compile your code to be aware of the Infiniband network. To do this run the following command.

% module swap openmpi openmpi/1.2.6-pgi

The mpicc, mpiCC, mpif77 and mpif90 will now make Infiniband aware binaries. You must load the openmpi/1.2.6-pgi module once everytime you log into nyx. You will also need to add the following line to your PBS scripts to ensure that the Infiniband-aware environment is exported to the batch system:

#PBS -V 

A code that has been compiled for Infiniband can still use a Ethernet network available on all nodes on nyx. This means that a recompile of already Infiniband aware binaries is not needed to run on nodes that do not have Infiniband. The mpirun included with the openmpi/1.2.6-pgi module will first look for the high speed Infiniband network. If it does not find it it will then use the Ethernet network. This means that you do not need to have multiple binaries for multiple networks. A single binary (once compiled to be Infiniband aware) will work on all nodes on nyx, taking advantage of the fastest networks first.

To force a job to require Infiniband nodes request the 'ib' property in your PBS script by using a line in your script like this:

#PBS -l nodes=4:ppn=4:ib,walltime=4:00:00