* Posts by PGrun

2 posts • joined 3 May 2010

Ethernet borgs RDMA from InfiniBand


IB and RoCE are not intended to compete

Alien Anthropologist is quite correct. IB has traditionally led Enet by several generations in bandwidth and latency and has always been the lowest price/performance.

That said, there are environments where it is simply not possible to transition to a new IB infrastructure. For example, many commercial enterprise data centers could benefit from IB's value propositions other than its raw bandwidths and industry leading latencies. For example, RoCE can bring major decreases in CPU utilization and reductions in memory bandwidith demand to an existing Ethernet infrastructure thus significantly reducing the number of servers required and reducing power consumption and footprint. These are the types of markets at which RoCE is aimed. RoCE does not compete with IB in terms of pure horsepower or price/performance. The fact that RoCE delivers 'IB-like' latencies is a very nice advantage.

Paul Grun

Chief Scientist, System Fabric Works, Inc.

Chair, IBTA RoCE Technical Working Group



What Tom says is true. iWARP is usually implemented in an RNIC which effectively offloads TCP into hardware. This is how iWARP is able to deliver its latency numbers. The drawback is that you cannot differentiate iWARP traffic from conventional Ethernet traffic until the packet has been cracked at the TCP layer. At that point, if the packet is conventional Enet, and not iWARP, it is effectively forked off to another process running a standard network stack. I would guess that this would make it challenging to run both conventional Ethernet (for networking and FCoE) and iWARP for IPC over the same Enet fabric although I believe the iWARP vendors are able to do so.

RoCE, on the other hand, is identified by a unique identifier in the Ethertype field. This means that Ethernet traffic is sorted and handled at the very bottom of the network stack, before any processing has been executed on it. Conventional TCP/IP packets are sent to a TCP/IP stack and RoCE traffic is sent through the RDMA stack .