Infiniband FAQ

Q: What is Infiniband?

A: Infiniband is a high bandwidth low latency network commonly found on clusters.

Q: How fast is Infiniband? What is its latency?

A: Infiniband is a 10Gbps network, all Infiniband at the CAC is nonblocking (no single connection is ever starved of the entire 10Gbps link. The latency of Infiniband is about one order of magnitude less than Ethernet. This puts Infiniband at less than 10 usec.

Q: Do I need to recompile my code to use Infiniband?

A: If your not sure which MPI library you used to compile your code yes you will need to recompile you code. See Compiling for Infiniband.

Q: How do I compile my code to use Infiniband?

A: To compile your code with support for Infiniband run:

$ module swap openmpi openmpi/1.2.6-pgi

This will update your environment. The mpicc, mpiCC, mpif77 and mpif90 will now make code that can run on Infiniband.

Q: I compiled my code for Infiniband can I run it on TCP (Ethernet) only nodes?

A: Yes! This is the benefit of the newer MPI library. Code compiled for Infiniband will also have the ability to run on Ethernet without recompiling. If mpirun finds that some of the nodes given to you do not have Infiniband then those nodes will use Ethernet. The nodes with Infiniband will use Infiniband.

Q: What advantages does Infiniband give me?

A: Infiniband allows bandwidth and latency sensitive codes to scale farther. For a example a code that scales to 12 cpus on Ethernet might get to 24 on Infiniband. If a code does not communicate often (like Monti Carlo codes) Infiniband will not improve your code's performance.



About the CAC Academics Getting Started Getting Help FAQ Resources Current Outages No outages have been reported Planned Outages No outages have been scheduled Search The CAC	Infiniband FAQ Q: What is Infiniband? A: Infiniband is a high bandwidth low latency network commonly found on clusters. Q: How fast is Infiniband? What is its latency? A: Infiniband is a 10Gbps network, all Infiniband at the CAC is nonblocking (no single connection is ever starved of the entire 10Gbps link. The latency of Infiniband is about one order of magnitude less than Ethernet. This puts Infiniband at less than 10 usec. Q: Do I need to recompile my code to use Infiniband? A: If your not sure which MPI library you used to compile your code yes you will need to recompile you code. See Compiling for Infiniband. Q: How do I compile my code to use Infiniband? A: To compile your code with support for Infiniband run: $ module swap openmpi openmpi/1.2.6-pgi This will update your environment. The mpicc, mpiCC, mpif77 and mpif90 will now make code that can run on Infiniband. Q: I compiled my code for Infiniband can I run it on TCP (Ethernet) only nodes? A: Yes! This is the benefit of the newer MPI library. Code compiled for Infiniband will also have the ability to run on Ethernet without recompiling. If mpirun finds that some of the nodes given to you do not have Infiniband then those nodes will use Ethernet. The nodes with Infiniband will use Infiniband. Q: What advantages does Infiniband give me? A: Infiniband allows bandwidth and latency sensitive codes to scale farther. For a example a code that scales to 12 cpus on Ethernet might get to 24 on Infiniband. If a code does not communicate often (like Monti Carlo codes) Infiniband will not improve your code's performance.