Infiniband (IB OFED) adoption inhibitors in Enterprise Data Centers


The following presentation explorers a list of inhibitors to infiniband deployment within enterprise data centers. Infiniband has limited adoption within Enterprise Data Centers. It has been relegated to isolated, special use cases such as HPC, or RDBMS that use RDMA within clusters. It’s high bandwidth and low latency make it ideal for these targeted use cases.

With increases in ethernet bandwidth and the introduction of ultra low latency switches, some ethernet pundits claim infiniband is no longer required in the Data Center.

Packet Latency – At this time, Infiniband stills maintains a small packet latency advantage over ultra low latency ethernet switches. For some applications, this small difference drives the continued use of Infiniband.

Network Bandwidth – Both Infiniband and Ethernet support high bandwidth levels. However, the Infiniband standard is designed to increase bandwidth standards faster then Ethernet. This has allowed Infiniband to increase bandwidth levels faster then Ethernet. With the introduction of high bandwidth ethernet, sales of infiniband switches has slowed. Consolidation of Infiniband vendors has reduced the market to one supplier (Mellanox). Mellanox now creates products for both Infiniband and Ethernet.

Cost Differences – At the time of this post, the cost of Infiniband .vs Ethernet switches is approx the same. Adapter cards are approx the same. In fact, we usually purchase network adapters that support attachment to both ethernet and infiniband switches to avoid the need to swap network adapters. Hybrid switches are now available that support both Infiniband and Ethernet (using Mellanox Switch-X ASIC). And finally, DAC and fiber cables for Infiniband and Ethernet are approx the same cost. The cost of deployment is no longer a major factor in the selection of a switch type.

In our view, three issues remain –

1) How HA is configure using Infiniband .vs Ethernet

2) Network staff skill level on the deployment, management and repair of Infiniband networks

3) OFED software stack quality

The following presentation explorers a list of inhibitors to infiniband deployment within enterprise data centers.

Download (, 499B)

It appears your Web browser is not configured to display PDF files. click here to download the PDF file. 

Leave a comment