Using Infiniband on nyx

There are a number of nodes on the cluster nyx.enign.umich.edu that have a Infiniband network available to use by users of nyx. Infiniband is a low latency high bandwidth network with speeds up to 10Gbps and latency of less than 10 usec. Infiniband allows codes that rely heavily on communication to scale to more cpus with higher efficiency.

To use Infiniband you must compile your code to be aware of the Infiniband network. To do this run the following command.

% module swap openmpi openmpi/1.1.0-pgi

The mpicc, mpiCC, mpif77 and mpif90 will now make Infiniband aware binaries. You must load the openmpi/1.1.0-pgi module once everytime you log into nyx. You will also need to add the following line to your PBS scripts to ensure that the Infiniband-aware environment is exported to the batch system:

#PBS -V

A code that has been compiled for Infiniband can still use a Ethernet network available on all nodes on nyx. This means that a recompile of already Infiniband aware binaries is not needed to run on nodes that do not have Infiniband. The mpirun included with the openmpi/1.1.0-pgi module will first look for the high speed Infiniband network. If it does not find it it will then use the Ethernet network. This means that you do not need to have multiple binaries for multiple networks. A single binary (once compiled to be Infiniband aware) will work on all nodes on nyx, taking advantage of the fastest networks first.

To force a job to require Infiniband nodes submit to the 'ib' queue you should request the ib queue in your PBS script by using a line in your script like this:

#PBS -q ib



About the CAC Academics Getting Started Getting Help FAQ Resources Current Outages No outages have been reported Planned Outages Aon – 07/26 Search The CAC	Using Infiniband on nyx There are a number of nodes on the cluster nyx.enign.umich.edu that have a Infiniband network available to use by users of nyx. Infiniband is a low latency high bandwidth network with speeds up to 10Gbps and latency of less than 10 usec. Infiniband allows codes that rely heavily on communication to scale to more cpus with higher efficiency. To use Infiniband you must compile your code to be aware of the Infiniband network. To do this run the following command. % module swap openmpi openmpi/1.1.0-pgi The mpicc, mpiCC, mpif77 and mpif90 will now make Infiniband aware binaries. You must load the openmpi/1.1.0-pgi module once everytime you log into nyx. You will also need to add the following line to your PBS scripts to ensure that the Infiniband-aware environment is exported to the batch system: #PBS -V A code that has been compiled for Infiniband can still use a Ethernet network available on all nodes on nyx. This means that a recompile of already Infiniband aware binaries is not needed to run on nodes that do not have Infiniband. The mpirun included with the openmpi/1.1.0-pgi module will first look for the high speed Infiniband network. If it does not find it it will then use the Ethernet network. This means that you do not need to have multiple binaries for multiple networks. A single binary (once compiled to be Infiniband aware) will work on all nodes on nyx, taking advantage of the fastest networks first. To force a job to require Infiniband nodes submit to the 'ib' queue you should request the `ib` queue in your PBS script by using a line in your script like this: #PBS -q ib