The future of PCIe: Get small, speed up, think outside the box

simplicity ... helps developers ... reap the benefits of PCIe in small form-factor devices.

What are the benefits, specifically?

grain of salt

PCI-e over external cabling has been on the market for a couple of years - for external interconnect to DAS RAID boxes and for additional PCIe slots via external expansion boxes, the latter sometimes combined with PCI-e IOV. Besides IOV, there are also simple rackmount switches to connect multiple external expansion boxes to a single "external PCIe socket" on a host computer. Ever fancied an industrial PC with some 30 PCI-e slots? Well it's been available for a while... As for PCI-e generations, in the external DAS RAID boxes I've seen PCI-e 1.0 and 2.0 (Areca). Even the connectors seem to be somewhat standard (no idea about their name). The interface between a motherboard (PCI-e slot) and the cable is in the form of a tiny PCI-e expansion board - interestingly it doesn't carry a switch, it's just a dumb repeater, or a "re-driver" as Pericom calls the chips used on the board. Apparently the chips provide a signal boost / preemphasis / equalization for the relatively longer external metallic cabling.

As for HPC: apart from storage and outward networking, HPC typically requires low-latency memory-to-memory copy among the cluster nodes. The one thing that to me still seems to be missing, for bare PCI-e to successfully compete against IB in HPC, is some PCI-e switching silicon that would provide any-to-any (matrix style, non-blocking) host-to-host memory-to-memory DMA, that combined with a greater number of host ports. IMO it wouldn't require a modification to the PCI-e standard: it would take some proprietary configurable swiching matrix implemented in silicon, providing multiple MMIO windows with mailboxing and IRQ to each participating host, combined with OS drivers and management software, that would interface to the HPC libraries, would take care of addressing among the nodes, and maybe provide some user-friendly management of the cluster interconnect at the top end.

The switches currently on the market can do maybe 4 to 8 hosts of up to 16 lanes each, and the primary purpose is PCIe IOV (sharing of network and storage adapters), rather than direct host-to-host DMA. Check with PLX or Pericom. Perhaps it would be possible, with current silicon, to do the sort of a matrix DMA interconnect in a single chip, to cater for about 8 hosts of PCI-e x8 or x16. That's not too many nodes for an HPC cluster. For greater clusters, it would have to be cascadable. Oh wait - that probably wouldn't scale very well.

As for PCI-e huddling with the compute cores: the PCI-e actually has an interface called a "PCI-e root complex" or a "host bridge" to the host CPU's native "front side bus" or whatever it has. The PCI-e is CPU architecture agnostic - and has some traditional downsides, such as the single-root-bridge logical topology. No way for a native multi-root topology on PCI-e - that's why we need to invent some clumsy DMA matrix thing in the first place. And guess what: there's a bus that's closer to the CPU cores than the PCI-e. On AMD systems, this is called HyperTransport - AMD actually got that from Alpha machines, but that was probably before PCI-e even existed. Intel later introduced a work-alike called QPI. The internal interconnects between the cores in a CPU package (such as the SandyBridge ring) are possibly not native HT/QPI, but these cannot be tapped, so they don't really count. So we have HT/QPI to play with: these are the busses that handle the talks between CPU sockets on a multi-socket server motherboard. Think of a cache-coherent NUMA machine on a single motherboard. And guess what, the HyperTransport can be used to link multiple motherboards together, to build an even bigger NUMA machine. There are practical products on the market for that: a company called NumaScale sells what they call a "NumaConnect" adaptor card, which plugs into an HTX slot (Hypertransport in a connector) on compatible motherboards. Interestingly, there is no switch, but the NumaConnect card has 6 outside ports, that can be used to create a hypercube or multi-dimensional torus topology of a desired dimension.

The solution marketed by NumaScale uses HyperTransport to build a ccNuma machine = it keeps cache coherence across the NUMA nodes. There's a somewhat similar solution called HyperShare that seems to use a cache-incoherent approach... Either way it seems that memory-to-memory access between the nodes is an inherent essential feature.

I've never heard of Intel making its QPI available in a slot. PCI is originally an Intel child, if memory serves. Maybe that's a clue...

Makes me wonder how much sense all of this makes in terms of operational reliability and stability. Are the NumaScale and HyperShare clusters tolerant to outages? Can nodes be hot-swapped in an out at runtime? One part of the problem sure is support for CPU and memory hotplugging and fault isolation in the OS (Linux or other) - another problem may be at the bus level: how does the HT hypercube cope when a node or link goes out to lunch? Makes me wonder how my theoretical PCI-e "matrix DMA" solution would cope with that (perhaps each peer would appear as a hot-pluggable PCI device to a particular host, with surprise removal gracefully handled). Ethernet sure doesn't have a problem with that. Not so sure about IB.

Sounds really good, so it won't be a success.

Nothing decent every finds its way into PC land, it has to be cheap and nasty like USB.

Jaw dropping

It is, for sure. Computing is just now starting to get really interesting. We thought it was all very exciting to go from 4mb memory to 32 mb for only $1,000 per 32mb stick, and from 33mhz cpus to 90mhz, but this kind of transfer tech that does 64 GB per second: amazing. It's hard to keep up with the changes, they're coming so fast.

Intel is dropping PCIe by 2016.

That's right, Intel is already shipping Oak Trail sans PCIe however the Federal Trade Commission has ORDERED Intel to continue PCIe support to at least 2016. You can read this in the Intel vs AMD settlement agreement.

Intel reasoned to FTC that mobile devices have no need for high speed interconnects. FTC "allowed" Oak Trail to be released but future silicon must support PCIe.

Without PCIe nVidia discrete GPU's do not find their way on Intel motherboards. Intel wants to go it alone without nVidia discrete cards. The believe that by 2016 on-die cpu/gpu soc will make the discrete gpu obsolete for all but the most demanding rendering. And I'm sure that Intel graphics MIGHT be up to speed by then.