Server SANs: Don't throw the baby out with the bathwater Getting rid of SAN complexity and cost is an idea that appeals to many CIOs sick of juggling storage area networking components and array costs. Let app servers virtualise a SAN into existence by pooling their own direct-attached storage and just buy servers with server SAN software, such as StoreVirtual from HP, DataCore's SAN …
Where is server SAN supposed to compete ? Somewhere between NAS and iSCSI SAN ? And these execs who are huffing about the "complexity" of centralized storage - are they supposed to be attracted by the exponentially greater complexity of distributed server storage ? Where is my file again ?
Distributed storage was great in the early 90s, when every node was also a data server. Until our fun was spoiled by Auspex et al. It was a Unix Admins dream and I'd love to see it come back. But it all came down to cost, mainly cost of support. 'Scuse be now, I have to go and do a Heartbleed firmware patch on my 47 heterogeneous storage nodes. 47 different procedures.
No, they sit in the SAN market (though yes all the ones I know of are iSCSI). It’s important to note that it is a distributed virtual SAN, NOT a distribured file system. So rather than servers with their own storage the whole lot is virtualised into a single ‘virtual SAN’. For example if you have 4 nodes each with 1TB in, it all gets mushed together by the software to give you a 4TB (raw) virtual SAN presented as an iSCSI target.
So yes the data is going to be spread over the servers, but at block level rather than “this file is on this server, that file is on that server…”. Naturally there’s support for things like multiple copies of data to survive one or more servers falling over. Rather than physical SAN controllers in the back of a physical storage box you now have a SAN controller running as a VM on each server node. That cluster of storage controllers works out where the data goes, where the copy goes, where it’s replicated to, what tier it belongs on and so on and so forth. There’s typically a quorum system in place so that a node failure doesn’t make the rest go nuts.
The general advantages are much like most virtual stuff – you can introduce new hosts pretty easily, you can add a new host, move data to it and retire an old host, you can add storage by both increasing the capacity of each node and by adding more nodes (which in theory also adds performance up until a point), typically speaking you can have multiple clusters and replicate between them for DR.
The actual capabilities are nothing new: to name two LeftHand did it for years and that product forms the basis of the HP StoreVirtual offering (which is now on generation 3 under HP ownership and I believe there were a couple of generations under LeftHand before that), and the Dell EqualLogic does something similar). The difference is that rather than buying a hardware appliance running this stuff, you're now buying the stuff to run on your own hardware and using the combination to create your own appliances. The news behind this news is simple that VMware are attempting to create their own pseudo-industry standard with the EVO:RAIL product, and more companies are jumping on the bandwagon hoping that you'll buy the whole stack from them.
Good explanation thanks. It is still not easy to see where distributed/virtualised SAN is going to fit in. The lower midrange storage market is pretty well sewn up by iSCSI arrays, NAS units and even FreeNAS, with its ZFS abilities. All have their advantages but one thing they have in common is cheap admin costs and minimal setup. Is there really a demand for yet another solution ? One with higher ongoing cost of ownership ?
Apologies if any of this reads funny or doesn't make sense, but it's late and I really can't be bothered to proof read it.
Anyway - it's not a one-fit solution but there's a few areas where it plays well -
1) Multi-site installations
Ability to spread a cluster across multiple buildings/sites, or have a cluster on site A replicate to site B either synchronously or asynchronously. Naturally spreading a cluster and synchronous replication between clusters have bandwidth and latency requirements but can be handy if there's two buildings on the same site - campus environments like schools, colleges, unis, hospitals, councils etc typically have that kind of setup.
The low end arrays (be they iSCSI or FC) typically vary somewhere between not being able to do replication at all or simply being crap it. You can often do something similar with software, such as storage clustering on Server 2012/Hyper-V, but it's typically not as good resulting in some data loss and downtime. As with everything it's a case of whether that's 'good enough' or if it justifies more investment, and the answer will vary by business
This isn't just for HA/DR purposes but can also be used for remote sites, for example having small clusters on remote sites replication to a larger cluster at a main site. .
2) Converged infrastructure
Excuse me while I go and cry in the shower having used that meaningless marketing bullshit term......back now. What I mean is that if you have a fairly small environment of say 3 VM hosts you can add a VSA licence to each host and whack some disks into each server. A small environment like that would typically only have a single low-end SAN like a HP MSA. No matter how resilient the VM hosts are with vMotion and affinity rules etc, if that MSA bites the dirt you're buggered. If you've got the VSA running on each node then the storage can stay online in the event that a node dies. Running the appliance does add a load to the host but compared to what servers can do these days it's pretty minimal (2 vCPUs, a dedicated 1GbE or 10GbE vSwitch and typically around 8-16GB RAM depending on the size and what features are in use).
This also plays well with the remote site argument. If you need local servers on your remote site do you want to implement a physical SAN on each site or just bang a VSA and some disks on each machine? Implementing a VSA on a couple of servers for a remote site with relatively low storage requirements isn't particularly expensive, it might even weigh up favourably compared to something like a MSA. Plus if you go down the physical route you're going to have poor DR capabilities with a cheap SAN, or a hell of a big bill for a better SAN.
3) Feature set
Virtual SANs typically include more features than low end SANs but cost less than the higher end physical SANs. For example the HP VSA (and the physical StoreVirtual appliances) include all features by default - all of the replication, clustering, multiple site stuff but also things like Adaptive Optimisaton* (HP fluff for automated storage tiering) and thin provisioning. Admittedly HP and others have brought tiering down into the MSA market but there's a cost involved, and features like thin provisioning are still usually the domain of larger arrays. And the cost of those features on larger arrays? You best sign that blank cheque.
* Before I get picked up on it by someone the exception is that the 1TB and 4TB VSA licenses don't include AO, however the 1TB licences are free and 4TB licences are something like 1,500-2,000 for a pack of 3. At that point you've got the capability to run a multi-node SAN up to 12TB for less than the cost of an entry level rackmount NAS, you just need to add disks (because you're not running your environment from an entry level QNAP with WD Red disks, are you?).
As for management it is no different to managing any other solution. Installation is easy - effectively you download an EXE, run it and point it at your host. Once it's done once you can point it at other hosts and tell it to use the same settings. Rinse and repeat. Takes a couple of minutes to bash in the settings and perhaps 15 minutes to deploy the appliance to a host. Once it's all set up you manage the cluster as a whole, rather than each individual node, so it's typically no more complex than managing a single physical SAN. You don't have to mess about changing a setting on VSA 1, then changing it on VSA 2, then on VSA 3...etc.
Naturally TCO varies on the environment so it doesn't work well for all scenarios. Split 4 VSAs between two buildings and the initial software bill is roughly 10,000 for 4x 10TB 5 year licenses. A pair of MSAs would cost around 12,000. Oh and the replication licences for those MSAs are about 2,500 each and naturally you'll be needing 2 of them. Supporting both of those MSAs after 3 years will cost perhaps 3,000 a year, so over 2 years another 6,000. In other words over the typical 5 year lifespan that we're seeing on systems these days in this scenario you would save 13,000.
I've not even put a value on adding a care pack to the MSA if you wanted something better than the basic next business day warranty during the first 3 years. The VSA has it's own support and the disks associated with it will be covered by care packs on the host servers, which you'd often buy anyway.
Even if you completely winged it with the base warranty and no post-warranty cover you'd still save the 7,000 up-front.
Of lower impact to TCO, but also true, is that server disks are typically cheaper than SAN disks (especially for SSDs) and also the power/cooling cost of running the VSA appliance on a few servers will be lower than running a pair of physical SANs.
Hmmmm HP-wise around £1,800 per node for a 10TB licence. So one of these hyper-converged 2U 4-node jobbies would have about £7,200 worth of licensing for the storage.
Again HP-wise the recommended configuration for most environments is network RAID 10. Much like disk RAID 10 that halves your usable storage. So 4 nodes each with 5TB presented to the VSA by the host will result in 10TB usable on the virtual SAN.
Because of that the bulk of the cost tends to be in the amount of disks required, rather than the licensing. That said what people often forget is that to do the same level of availability on a conventional SAN you'd need two SANs. Funnily enough that means doubling up the number of disks, and typically you'll end up buying things like replication licenses as well.
Not entirely true with a VSA and dual copy strategy if you lose one half of the mirror you no longer have H/A, so in reality you need three copies, more expense, more complexity. With dual SAN's you lose DR but keep H/A even if you do lose one.The other thing that everyone assumes is that all of these copies are for free, and I don't mean capacity, I mean the need to distribute the backend IOps and the subsequent latency incurred for the writes over the Network.
Sorry, not getting you on that one? In that if you have two nodes of something, be it physical or virtual, and you lose one of them you're in a pretty precarious position until it's back online. Also please remember that with these virtual SANs we're in the price range of things like a HP MSA or a Dell PowerVault, not the higher end storage systems that genuinely can do HA and DR properly. If you count replication on systems like an MSA as HA then I hope to god you never have to test it.
IOPS and network traffic are undeniably a problem when going up to a large number of nodes, but that's a flaw inherent with any scale-out system of this nature, virtual or physical. At that scale of system it's something to consider when deciding whether the scale out system is going to network itself into oblivion and it's better to recommend something like a 3PAR, Compellent, EMC etc.
PS - Just in case we're crossing wires, when I say VSA I'm talking specifically about the HP StoreVirtual VSA which functions differently to VMware vSAN.
The only option for data protection in VSAN and most other "Server SAN"/"Distributed DAS" approaches is mirroring, preferably a triple mirror. This creates much higher raw/usable ratio compared to parity RAID on external arrays.
This will also drive up power, space, and cooling requirements per usable GB.
The advantage of Server SAN/Distributed DAS is cost. EVO:RAIL requires the use of higher $/GB SAS disks instead of cheap $/GB SATA disks. Nutanix seems to be moving towards SAS solutions as well. However, the $/GB of server SAS HDDs is competitive with the $/GB of disk array vendors' SATA options. The same effect will likely happen with eMLC SSDs at some point. But all-flash array vendors might have the upper hand with their parity RAID, data reduction, and consumer grade flash enabled by aggressive write management, so the era of Server SAN might be limited as most on-line storage moves to all-flash over the next few years.
As for the idea of a local copy of data on the server, that was the idea behind EMC's Project Lightning/VFCache/XtremCache. But with other in-memory and server flash solutions (i.e., Flash DIMMs) gaining traction, big memory servers may become the norm, so local storage may become less relevant from a performance aspect.
Of course, if you're relying on a single SAN and that SAN buys it, you're pooched. Server SANs can be configured to support multiple node failures without giving up the ghost.
Just use two SANs in block level replication to one another? Let's rerun the numbers...
You can run 2 x Synology in HA block level replication over a dedicated link. Combine with a load of off the shelf disks including the new range of endurance rated SSDs and a few cheap all-port 10GB switches from Netgear with dedicated 10GB links on the servers and you could see performance that beats a lot of SANS in current use including FC.
Cost wise, especially without per TB licensing you get more for less. You can also add new storage nodes as you speed/capacity requirements changes, very quickly. You may find adding a new server node a bit more trouble.
All running well Vsans seem fine. When things start going south with no definitive cause and you are trying to work out whether it is the VM, the server, a remote node etc there is something that is quite nice about a separate system - processing on one side, all storage on another.
"You can run 2 x Synology in HA block level replication over a dedicated link"
Okay, sure. I do this all the time. I still have several configurations where server SANs are far cheaper, especially for a given required performance level.
Neither solution is all thing to all people. Not yet, anyways.
Substantial use of the word 'could'. It's OK to have some innovative thinking sometimes, but there are far too many guesses in here.
I'm not sure what problem 'server SANs' are really solving. From this article it seems that they are unable to deliver data fast enough to servers because of the network time involved in the data transfer, and so they would seem to be unsuitable for their core purpose - supplying data to apps. Suggesting that data then has to be cached in the server increases complexity just to cover the inadequacy of the chosen storage platform - chosing a fit for purpose platform is a much better solution.
I'm not a fan of big arrays, but this seems to be a retrograde step even for them. They are built for online disk replacement, multi-tiered caching, fast(ish) FC connectivity etc etc. moving to an estate of generic 2 CPU servers with a bunch of disks attached to each somehow seems a retrograde step, and IMHO for certain a maintenence nightmare.
This glosses over the real storage challenge - how the hell do we server up data fast enough to the massive amounts of CPU throughput we are seeing on even 2 CPU servers. There's no way that even a pair of 10GbE connections is going to be enough, so unless we rethink the core datacentre network (and physically upgrade) then server SAN's seem to be a solution looking for a problem.
"From this article it seems that they are unable to deliver data fast enough to servers because of the network time involved in the data transfer, and so they would seem to be unsuitable for their core purpose - supplying data to apps."
Yeah, I really don't know where this comes from, TBH. You're going to have to go a long bloody way to convince me that server SANs can't deliver the IOPS or have any worse latency than arrays. What I get out of infiniband systems...well, there's reviews upcoming on that, naturally.
Am I the only one that seems to think running my storage system as a VM ontop of the very system that is consuming it is a little scary not to mention performance limiting. If the Hypervisor is going to dish out CPU and RAM resources then it should dish out SAN/Storage resources.
Secondly with Server SAN where do I fit my low latency 40TB database?
I think the real answer lies in something like ScaleIO hears why:
Have servers contributing CPU/RAM
Have servers contributing storage
Have servers contributing CPU/RAM/Storage
need to scale any one part scale it; need to scale all parts scale them; but this notion of "web-scale" scale them all together does not work when you are not "web-scale" (meaning I run a very distributed application or lots of small VM's) and if you are then enterprise hyper-visors is not the solution you will be using if you expect to turn a profit.
Once you are in the game of scale any part independently you then start to look at what parts you can swap out as no one wants a datacenter with all verticals tied to a single vendor as that is total lockin. You come back to a traditional SAN again.
The argument is mute as the entire reason this all exists is due to the costs of storage and the slowness of disk. Force storage vendors to use commodity (vendor or customer supplied) hardware and store on flash and we get what we have been asking for all along do for storage what Windows/Linux did for servers.
Image a storage platform where services could be added by installing a plugin or extra software or even a docker container from any vendor and then using it.
We see this now happening on the Network side and I think it is where storage will go.
Can anyone tell me what you actually gain from server SAN vs. a modern array? I know my numbers have shown I can get an entire blade chassis full of very high spec blades and a Pure Storage 420 50TB SAN for about the same price you would get a modern converged system all be it of smaller capacity.
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
