Interesting article Chris. I've got my money on one of these formats becoming more pervasive into the datacentre for large scaling non x86 systems. Wanna wager? :)

The PCIe concept is intriguing, I can't see it working at a director level though for storage. It's hard enough now to look at throughput on huge servers running multiple partitions at how hard the interconnecting buses are running, there's literally no tooling for some platforms for bus utilisation. I mean a switch has an ASIC for the traffic, the HBA obviously is an ASIC on a board to pull/push the traffic that hard to/from it's bus. Doesn't this mean that the switching kit inside a switch would have to have a lot more throughput?

Extend that out to storage and sans and I'm wondering what tooling they could come up with, though I imagine it'll lose some latency because you'd be negating the need for an HBA or a NIC. Maybe this would work for small hosts running a single HBA or two but we have some lpars running 20+ HBA with a processor footprint probably having 60+ HBA across 10 high speed busses to the processing unit.

Hmmm I need more coffee. My head hurts.

a start-up is redefining a mainframe... or minicomputer... just with external servers and blade racks?!?!?!

Im pretty damn sure that Big Blue's AS/400 can very happily run multiple instances of virtualised servers (linux based too....) across one set of internal I/O hardware...

So what exactly are they bringing new to the environment?

It's no wonder they have no product... they can't afford the patents or cross licensing needed for a "mainframe", at least not without potential customers saying "then why don't we just by an IBM"!

{stupse}

Why PCIe? If we go that route we might as well look at hypertransport (or its Intel analogue). Makes much more sense. In that case the rack of servers is not just using virtualised IO. It can use both virtualised IO and memory and be an arbitrarily big NUMA system.

Granted, OS designers will scream for a while, but overall, NUMA scheduling, memory allocation, etc have answers now and most OS-es can support it properly. In fact multicore+multisocket or multicore+multithread is a form of NUMA already.

with the POWER5/6 frames and Virtualisation?

We've got over 40 p595's alone with 64 CPUs/1TB RAM each and 4 * D20 I/O drawers with 24 PCI slots each... all shared virtually where possible, or dedicated where extreme throughput is needed. These host LPARs with virtual processors and allocated memory... job done!

Very, very interesting.

If you think about this along with all the other consolidation that's going on, it's almost redefining computers. You've got proposals knocking about these days for shared power supplies in datacenters, shared storage already exists, and virtualization (shared processors) is massive. This however is a whole new level, it's shared i/o, and potentially could apply to pretty much everything else.

If this takes off, each server can potentially be reduced to just processor, memory and motherboard. Absolutely everything else can be centrally provided and shared as needed. Each rack or server room becomes its own little mainframe / blade system.

It's also interesting when you think how well this could work with virtualization. The concept of mapping PCI-e to virtual machines has already been raised. Personally I think PCI-e mapping could be an easy way to provide high performance graphics to virtual machines. If shared PCI-e arrives and shared graphics cards (such as Nvidia's range) take off, there are potentially even higher gains to be had.

So if you think about it, we now have efficiency gains either in place or being talked about for:

- Shared CPU+ram (virtualisation)

- Shared storage

- Shared power supplies

- Shared networking

- Shared graphics

Combine all of those, and you can fine tune every aspect of your companies IT assets, precisely matching your hardware to your requirements. And you can do this while increasing performance since everything is connected in your server room using a local high speed interconnect.

Wow.

Sure you can rip out the HBA/NIC but there is not a External connector for PCIe for them to plug a cheap cable into each motherboard. You will still need a PCIe Card (a network interface card if you will) that will plug into the motherboard and provide a port for the shared resource to connect all the motherboards (servers) together.

I don't see this as either a energy saver or a big cost saver (especially since these proprietary External PCIe NIC will be more costly than a generic HBA).

However, the article should have focused on the real win here 8000 - 16000MB/s speeds which beats the pants off of even Gig Ethernet by a a fair stretch. The I/O backbone for this rack of servers will be fantastic. This would especially help for a bunch of servers that need frequent or large communications with each other.

I do wonder how the PCIe bus will scale though when shared amongst so many resources.

I know most MB offer more than one PCie 8x or 16x slot, but more than one card is used, many of these motherboards have to actually run at slower speeds due to limitations.

@Nathan Billet

What they are bringing new is the same shared architecture of a mainframe but made up of lower cost generic parts. No need to shell out $$ to IBM for parts and service, just have your local tech slap in a replacement MB, with newer faster CPU and RAM. Replacing proprietary parts with generic market is not new either, but it has been successful when done in the past (this is how Juniper jumped into Cisco's hot tub and crashed the party)

Many servers don't max out their nic or hba so pooling them would be advantageous.

VMware running on blades with a backplane can help, but PCIe is a more open standard than most blade-server backplanes.

Dual nics/adapters are often used for redundancy. By pooling the nic's/hba's you could run N+1 rather than N*2 adapters and still have a nic/hba per server and a spare available (virtualised so any host can access it). For disk access, multiple PCIe buses would be much cheaper to put into a server than multiple infiniband controllers and much more efficient than using nfs/udp/ip.

It seems like a good idea to me, though the thought of some techie pulling the wrong PCIe cable fills me with more fear than having them pull the wrong network cable!