Currently most CAC resources are tied together using a Force10 E1200 and three Force10 C300s, providing non-blocking gigabit speeds at normal ethernet latency.

We have a small Infiniband installation on nyx (9 nodes, with more to come) that is available for testing. Infiniband provides user-level, one-way data rates with a maximum theroretical bandwidth of 10Gb/s. On the nyx nodes the maximum measured bandwidth is about 6.5Gb/s, or about 7x faster than Ethernet. The measured latency of the Infiniband network is about 6us, or 10 time faster than Ethernet. For more information, see Infiniband on Nyx and our Infiniband FAQ.