Server virtualization – what could possibly go wrong?

Virtual Security

I'm glad that you're going to look at security, because it is often overlooked in the rush to virtualise; while the points that you've made about 'virtual server sprawl' highlights that there are risks, no least of which is the potential for forgotten test applications lurking unpatched and unattended in the virtual environment. Additionally, having a coherent and manageable way of managing the real and virtual environments is critical. We have the good fortune of working with Stonesoft who have developed technologies that secure real and virtual environments from a single, centralised management console, and we are very aware of how unusual their offering is.

dp

security/bugs thought

first off, thanks to the people who answered my last questions, pointing out that there was more than consolidation to consider.

However I've been concerned with possible interactions of VM behaviour with software requirements, esp. databases in my cases. They must flush stuff to disk in the right order and that must be guaranteed by hardware (not doing so leaves a window of possible data corruption, which is very small if running on well-managed hardware but it's still there, and not all hardware is well managed - I think we've all seen critical data on servers shoved into cupboards with no UPS and/or no firewall).

I finally found a statement on the VMWare site which said their bare-metal hypervisor would maintain exactly the same disk characteristics (can't remember the exact phrase) so at last I had an answer - but did anyone else even realise there was a question? I don't think so, and the question as to these characteristics on non-bare-metal hypervisors is still open (anyone running a DB like this? I'm curious). People are buying into this without being critical enough. I'm not saying VMWare is inadequate, just that it's early days and people need to be less accepting.

Which brings me onto another question regarding actual bugs. This is the third time I've posted this: <http://communities.vmware.com/message/967522> "WARNING: VM bug & SCSI disks causes file data corruption (SQL2005)!" (this is reported in workstation, not server. It's not clear whether it's been resolved).

A technology would have to be very, very mature before I put critical stuff on it. VMWare is damn good but I don't know if we're at that stage yet.

An aside. A respondent mentioned that exchange & sql server on the same machine would fight for memory. SQL server's memory is trivially controllable via gui. This set me off wondering if just getting familiar with their packages would lift some of the problems people are having. I've seen so many headaches because people just didn't know what their stuff could do.

eggs and baskets

First of all, virtual machines are nothing new. They've been the main, if not only way that mainframes have been managed for years, or decades even. While it's very easily just to click a new VM into existence, the mainframe world has never had any major problems with VM proliferation. This could be because the world opf mainframes is one of slow, considered administration based on a stable systems architecture. The mainframe world also comes with the tools to manage and document VMs, to ensure everyone can see exactly what's going on.

The main danger posed by virtualising everything is not understanding the risks. Running 10 VMs on one host means that if (when?) that single piece of hardware blows a fuse then you haven't just lost 1 service, you've lost a whole bunch - so the hardware reliability drops by a factor of the number of VMs each box is hosting.This is not so much an issue for mainframes, where a hardware fault is quite rare - but for a commodity level server, built down to a price-point the chances of the lights going out are higher and not well understood.

It's not just the physical hardware either. Consider other shared components such as the network. If the cleaner trips over the 1000Bt that's carrying all the traffic from all the VMs, then they too all fail at once. Same goes for storage connections too. All the reliability risks get multiplied.

Talking of which: a secondary risk, that's been in play for a long time now is an effect we see on storage arrays - people (managers, mainly) having no appreciation of how the users of shared resources interact with each other. All too often the link between allocated resources and physical hardware is too abstract, and consequently is viewed as simply a bottomless pit. I've seen situations where a production database slows down terribly for no apparent reason. After a *lot* of investigation it turns out to be because another business unit in another building has mirrored their backup staging area onto someone's production spindles. All completely different volumes, of course. However no-one made the connection down at the metal level. I can see much more of this sort of problem occurring in cloud-computing land, where no-one really knows which physical resources are being used by which production systems.

@Pete 2

They're all good reasons Pete, and to be honest there were quite a few of them in existance at my current place of work; however propper planning, management and implementation negate the vast majority of the problems raised.

Network and storage connections.... On a critical box I'd always used teamed NIC's anyway, on our ESX servers we have 4x1Gb pipes bonded with LACP with fully managed and carefully routed cables to a redundant core switch. Using a FC SAN I'd never put bare fibre onto a server. Again you'd have redundant paths (2 HBA's pointing to 2 FC switches in a mesh config), use armored fibre and ensure it's wrapped in a protective shielding.

Storage is interesting, but again it's all down to planning. If you have your DAE's partitioned into multiple LUN's using various RAID levels as appropiate then you at least know which LUN is being used by which VM guest, and can easily see what other VM's are using it as well. That would be the same as in a physical environment if you are using a SAN... just the nature of the beast. Propper planning, so you know what resouce your likely to need and not overloading the LUN's prevents these issues. Virtual or not.

Although I can see exactly where your coming from. We run our Citrix farm within VMWare which spans three server rooms across a 6 mile square area. (Factory) Should a user have a problem he could be on anyone of the 6 Citrix Servers, which in turn could be on any VM Host. And of course without looking all I know is that the physical disk is on one of 6 DAE's!!! (And I'm now implementing HA on our firewall and leased lines which means "virtualising the firewall"...!!!)

Putting all your eggs in one basket isn't a good idea normally, however if you can mitigate the risk whilst keeping the benefits (consoldation, management and cost) then it seems silly not to take that route.

@boltar

There are plenty of reasons why you might to choose to virtualise UNIX (or any other OS) rather than co-host applications. Firstly there is the issue of housekeeping and patch management - anybody who has worked on a co-hosted environment in a service-critical environment will know about the problems of co-ordinating outages for things such as patch management, introducing OS upgrades or some configuration changes that can only be done by bouncing the machine. Yes, there are times when the Hyperviser needs such treatment, but with the ability to move VMs dynamically, then downtime for that can be reduced.

Then there is the issue of isolating problems. Unix is actually not very good at that - a badly behaved application can bring the whole machine down. Anything from filling up swap, to forking to many processes can, even if it doesn't crash the machine, bring throughput to a virtual crawl. Anybody who knows IBM mainframes will know that there are far tighter controls workload management, but that's a far more rigid and less fluid environment. UNIX, Linux and so on are simply not like that - it's a strength, and it's a weakness at the same time. Doing your workload management at the hypervisor level can allow you to greatly limit the impact of badly behaved applications. Then there is the convenience fact of facilitating consolidation - yes, you might consolidated a dozen Unix apps onto a single co-hosted environment, but then you've got to sort out all those version and library differences, the clashing naming standards, the shared configuration files, the kernel settings. It's often more trouble than it is worth.

Also. virtual machines work particularly well in development and test environment. VMs can be bounced, libraries changed and the like without impacting everybody else.

Now this is not to say that co-hosting doesn't make a lot of sense. However, I'm much less convinced about co-hosting unrelated applications. For larger organisations it often makes sense to develop consolidation strategies that allow you to present "appliance-like" services. It's perfectly feasible to produce farms for co-hosting databases, J2EE environments, web-services. In that case you have virtualisation at a different level - that of software services. It's much more efficient in hardware utlisation terms than having a VM per application instance. You can then have an environment which is optimised for running a given type of workload.

Eventually is seems likely that everything will be virtualised by default - look at the mainframe arena. However, that's not instead of co-hosting, it's as well as.

As for my main problems with (machine) virtualisation. Firstly there's config management, especially insofar as it affects software licensing, management, performance and capacity planning. If you are going to move your apps all over an ESX farm you had better have a way of dealing with all those issues (and the software licensing one can really bite you in the backside - there are plenty of companies out there that will seek to optimise their revenue through their licensing models that con't recognise the reality of virtulaisation). Then there is the support problem - I've lost count of the number of suppliers that don't support virtualised environments. Some of it is FUD, some of it is real. Then there is the issue that machine virtualisation can be inefficient and ingrain bad practice. One thing that VMs do is chew up memory and disk space as every OS carries a considerable overhead of both. One of the major problems is that none of the OS's that you are likely to run on VMWare and the like will share code pages. For those that have used CMS on VM, that was specifically engineered so that different instances of the guest OS would share code pages.Not much chance of that with Windows or Linux (unless things have change, IBM gave up on doing the same with Linux under VM, but if it were possible it would save huge amounts of memory).

@boltar

@boltar, I agree. I think VMs are much more exciting for windows shops (due to Windows architectural flaws essentially). But there are cases where perhaps different divisions within the co want "their own" machine where VMs could be useful. Although, chroot jails, and other kernel-level variants of this, make VMs not so necessary even in this case.

The nice case would be if there's a failover setup on it -- so a machine acting up, the VMs automatically transfer. I think this is not commonly set up though outside the mainframe world (and perhaps Suns).

LPAR Sprawl on POWER

We have seen a lot of LPAR sprawl for our POWER-based systems. We migrated 30+ Sun servers to three IBM POWER5 boxen running 30 AIX LPARs, and in just a few years we have grown to over 100 AIX LPARs runnng on three POWER5 and two POWER6 boxen. We could have never afforded to do that many LPARs if we ran only one OS image per box.