A large internet service provider said data for as many as 100,000 websites was destroyed by attackers who targeted a zero-day vulnerability in a widely-used virtualization application. Technicians at UK-based Vaserv.com were still scrambling to recover data on Monday evening UK time, more than 24 hours after unknown hackers …
Seems like this has been known about for a bit but they don't seem to be doing anything about it!
Christ on a bike
That's not nice.
Even if the service is unmanaged do they not run backups as standard incase of hardware failure ?? I would be a bit miffed if I was a customer there.
I run a vps (cpanel) for some smaller clients & testing think I'm going to check my backups (2nd raid disk) and pull a set down to my local servers "just in case"
There really are some evil people with no fathers out there who probably also have carnal relationships on the maternal side
I have (had!) a backup server hosted by these folks... I used to have some more important stuff there, but pulled it out a few months ago because HyperVM was making me nervous. They pulled the entire control panel down several times since recently due to suspected vulnerabilities in the software.
Basically, HyperVM looks like a house of cards so I think it was only a matter of time before it got hacked. The control panel appears to run as root on each VPS host, of course any outward-facing thing can get hacked but there ought to have been some level of abstraction between the control panel and the VPSes to slow down the hackers. Doesn't seem like there was though.
Pretty glad I moved my stuff when I did.
I was affected by this hack. I am still awaiting a response.
A server which was hosting a very large campaign website was one of the servers hosted and attacked. I am still waiting on VASERV to issue an update regarding my node... although so far they have dealt with the matter well.
What has happened is a serious criminal act and those involved should be brought to justice. I am surprised this has not been mentioned in the mainstream news.
This is a major wake up call.
So Pissed off
We did have backups of the majority of the sites - but so many have been lost / backups were out of date. VAServ should be ashamed that this happened. I was one week away from moving everything to VPS.net aswell.
they cost good money for a reason. Hopefully they'll learn to secure their servers better next time... Or at least others using this platform. I am sure this company is going to get a _huge_ flood of customers after this....
Lack of information from VAServe
I have an unmanaged VPS that has disappeared. I feel sorry for VAServe, other than choosing poor management software, they couldnt have predicted this! The VAServe status page is a bit sketchy, and i dont know which physical server my node was hosted on, so i have no idea if all my data has been lost, or when it might be restored :-(
Not zero-day at all?!
If find the use of the term "zero-day exploit" a bit rich as(according to http://www.milw0rm.com/exploits/8880) the vendors were notified on the 21st of may and the exploits has been 'in the wild' for a few days now. Strangely enough I have a VPS from these guys and it has not been affected, perhaps to do with my server being xen based....
Absurd . . .
"Low-end" customers get no backup service? Since it's a VM-based structure, surely one day a week, the "low-end" accounts could be taken offline, and the directory holding each VM's files be ZIPped or RARed?
I used to run some shell boxes in VM for people, and would, upon request, backup the contents for them - even though they paid me NOTHING for the service, nor for the backup procedure. Backups were password-protected and placed in a private FTP directory for the user to "collect" within a set period of time. if the user chose not to download their backups, they were copied to an external HD, as well as burned to DVDrom for safekeeping. I once actually snailmailed a user's backups to him, as he only had a dialup connection and as such wasn't up for the large download from FTP.
Mind you, my service was a hobby, and was only accessible to a select few "trusted" friends. It hosted no "sensitive" data. It was not "important", nor was it ever advertised.
Tux, because... WAAARK! (or whatever noise Penguins make).
@Christopher Ahrens: They DID secure their servers, LXLABS (and their crap software) had a exploit which basically allowed root access to anyone running HyperVM.
You can find information here: http://www.webhostingtalk.com/showthread.php?t=867100
I don't think this is the fault of Vaserv, their software was fully up to date - I don't see why they should be ashamed as they take security very seriously.
Their status site is here for anyone that needs it http://vaserv.com/
if you look at the exploits on milw0rm, its laughable that lxlabs have even managed to string a product together. Im not a web apps expert but even i couldnt put out such shockingly bad software
You get what you pay for
More info here:
I host a number of VPS's with vaserve, their communication has been pretty good through this. They are a no frills host and it's pretty clear that if you want backups, you need to make sure you get backups.
Now replace web services,
With cloud managed servers, and you have another reason cloud is a bad idea (tm)
No kidding ?
I think we'll have to wait a bit for the UK Glovinmint to wake up. Possibly until the Cabinet return from their jollies, after the summer recess.
Re: Real Hypervisors
@ Christopher Ahrens
HyperVM is not the hypervisor, it is a web based management tool for Xen and Virtuozzo which are both "real" virtualisation technologies.
The real culprit here is LX Labs for not delivering a secure product, despite their claims on their website:
"We take security as the most serious of the concerns and have worked hard to create a secure environment where you can be confident about the server's state."
This is clearly rubbish.
eDarwin in action. The web could do with a bit less shite on it.
Now I know why
We haven't virtualized anything important in our environment.... poof ! So guys, where are those virtual backups ?
I've had swift responses but it shouldn't have happened.
Of my two VPSs one has been unreachable for over 24hrs. :(
Over the last year I've had a good experience with Vaserv and consider them quite low on bullshit. Even today I've emailed them twice and had replies in 2 minutes and 15 minutes. That is VERY fast given the scale of the problem they are facing. Obviously this whole situation shouldn't have happened and I'm facing data loss. However I could have bought the add on back up from them, used the web based control panel to do it or used rsync. So I can't really blame them for the lack of a more recent backup.
Initially they did a good job of keeping the information flowing. But then they listed some damaged nodes and then claimed "Everything else should be up and running for the UK". This was a mistake because it seems to be inaccurate, it raised expectations and I imagine it flooded them with queries about why things weren't working. Tired people make mistakes. Anyway hope this resolves as quickly as possible for all our sakes. I'm not very happy so my staying with them depends on their response to this exceptional disaster.
Not really a surprise
I'm still waiting to find out if my VPS was one that got rm'd, but if I'm honest I half suspected (along with some others here) that this might happen one day with HyperVM. Not because I knew anything about HyperVM itself, but because web apps in general pose so many security problems. There are usually many different input methods, all filtered differently and usually all with access to the crown jewels.
The annoying thing here is that if my VPS comes back on line do I assume the hacker(s) left a back-door somewhere, and rebuild it from scratch just to be safe? How very tiresome.
Paris because I saw this coming, ahem.
oUCH. Cloud Computing has a couple embedded CumuloNimbi. Kind of bad when a management tool can enable so much virtual destruction. Where's the shadow copy service that allows you to take hot backups for this? At least stuff would be a little out of date, but not totally gone.
"The researcher than found the vulnerabilities in the only gave the developer 2/3 weeks to fix it, looks like to me that he got peed off and released them to the public because lack of response."
I'd like to clarify this. First, I did not give the developer "2/3 weeks to fix" the issues. In fact, I did not give any timeline at all. What I did is what the advisory says. Had the vendor looked at the issues (which was not done) and requested some time to address them, of course, I would have given any amount of time requested before going public with the information.
Second, what it looks like to you and what really happened are 2 different things. I did not release the information out of anger with anyone. It was released so that customers, both current and potential, would be aware of the issues. It is not the job of the person who spends their time finding and documenting the bugs to babysit a vendor, plain and simple.
Third, as it stands, there is nothing whatsoever that definitively connects the current situation with the afflicted webhost with the information that was made publicly available. I audited Kloxo. As I understand it, and do correct me if I'm wrong, but they believe the issue was with HyperVM. I did not find out until later than HyperVM and Kloxo (formerly LxAdmin) share some of the same features/code.
Finally, lxlabs/kloxo/hypervm has been getting hacked for a while now, well before I ever published anything. Read their forums and you will see.
they must have pissed off the wrong person. Hacks of these styles tend to be quite personal.
VAServ - very good
I'm a customer and I'm very happy with VAServ - they are a budget VPS provider who offer a no frills service which I use for personal stuff / development. The poster above who claims to be hosting a major campaign with them is clearly an idiot.
In reply to the above, here is an excerpt from that advisory:
# Timeline :
# 05/21/2009 - sent initial email to vendor with a link to a private resource for viewing various kloxo hiab575 vulnerability info
# 05/23/2009 - received the following: "Thanks for the info. I will review this and let you know." (no signature)
# 05/30/2009 - sent an email asking if there were any updates
# 06/01/2009 - received the following: "Sorry for the delay. I am currently looking into this, and will reply in a couple of hours time." (no signature)
# 06/04/2009 - nothing heard from vendor, and the private resource containing the vulnerability info still does not appear to have been accessed
# 2 weeks have passed since the initial notification. Vendor appears uninterested.
Your qualification to call yourself a white hat is based on the pretence of being interested in giving Kloxo information about weaknesses that they could fix privately. In fact, you failed to give them reasonable time to respond before calling up the hackers. There's no defence.
We use(d) HyperVM...
We're a UK and USA based provider who have had to cease all new orders and remove HyperVM from all our servers.
After running several tests on development hardware, the issues still remain -- LXLabs have essentially packed up and gone home.
Bill gates, because he loves hackers.
"you gave people the warning to make a back up"
Where? He didn't even warn the company of what he was about to do. That would have woken them up.
The "full disclosure" cant is just a fig leaf in this case. I agree he can't be expected to babysit these stupid lazy tw*ts at Kloxo, but at the same time he overstepped the mark by a long way in this case. Knowledge gives power and privilege, but where was the _noblesse oblige_? He should learn from this, unless he just wants to create problems for himself in the future, especially given the dubious legal situation.
I am a small web host, that had primary services hosted with VAServ and I expect that if I am not on-line within another 24 hours, I will have to dissolve my company.
I also know another person who's entire hosting infrastructure has been compromised and taken down by this attack.
I am surprised that they did not mirror data on their servers, and do understand that the challenge they face now to restore that many websites and servers. Is indeed no small job.
Did he piss off the wrong clients?
Vaserv subsidiary a2b2.net has something of a history of providing IP space for phishes and spam/phish maildrops, and as recently as last week was running a site offering turnkey phish kits from the same IP address that had recently hosted several bank and PayPal phishes:
Wouldn't surprise me one bit to learn that the attack was perpetrated by a current or former customer, somehow.
Why are so many people moaning about lost data and then no idea where their backups are? For flips sake I have cruddy little personal webpage on some free provider with photos of me doing stupid stuff, but hell's teeth even I keep a backup of it, so it can be resent! I know you pay for services, but quite often you will find that cut-rate comes at a price.
If you have to trust someone else, DON'T! As an idiot manager once said to me, in one of his many David Brent moments, "We never assume, we always make sure. Right?".
Umm... Leet Speak for ???
It's all in a name
More disasters caused by Micro$oft...
M$ really should take the blame for the disasters they cause, with these zero-day bugs infesting their crapplications.
Oh? Sorry? This wasn't Windows-based applications?
Oh, right, I'll stop ranting then.
They should have used OSX. No security holes in that, no sirree!
But seriously, a total bummer to everyone who was on it....
<blockquote>I am surprised that they did not mirror data on their servers, and do understand that the challenge they face now to restore that many websites and servers. Is indeed no small job.</blockquote>
The neat thing about mirrored data is that valid data manipulation such as deletion of unwanted data gets backed up to your backup system.
Mirroring data is only valid for guarding against hardware failures that don't cause bad data to be stored and replicated. Unless there is versioned backup to offline storage, there really is no backup at all.
If you don't have regular backups, then you should know you are rolling the dice. If it wasn't this, then any number of other possible events could have taken out your data.
Seems like a lot of people learn this the hard way by losing a lot of data at some point in time.
If you used a hosting service without a backup plan, and then didn't create your own backup plan, you really set yourself up for this kick in the teeth.
I feel a disturbance in the intertubes...
....ah, 100,000 sites screaming out from the void....
If your business is reliant on your site/software.
I have only 3 things to say:
"I am a small web host, that had primary services hosted with VAServ and I expect that if I am not on-line within another 24 hours, I will have to dissolve my company.
I also know another person who's entire hosting infrastructure has been compromised and taken down by this attack.
I am surprised that they did not mirror data on their servers, and do understand that the challenge they face now to restore that many websites and servers. Is indeed no small job."
If your time line is only 24 hours from when you posted I take it you made sure your SLA with them for restoration of service was well within this time-frame when you signed up?
I doubt you did and I can have very little sympathy for someone who appears to be sent under by this because they failed to address the issue of tail events and business continuity.
I hate to break this to you, but it's sort of your fault for relying on one provider.
Even if you're a low-end provider (hosting sites on someone else's VPS setup is low-end IMHO) it's not hard to get at least one box somewhere else and regularly smuggle your data back and forth. Having all your boxes at one provider is stupid for all sorts of reasons what happens to your DNS, mail etc when the provider has down time, or goes bankrupt and gets their cables pulled?
The CLOUD is Open and Virtually Closed for Banking Business.
"If you have to trust someone else, DON'T! As an idiot manager once said to me, in one of his many David Brent moments, "We never assume, we always make sure. Right?"." ..... By Anonymous Coward Posted Tuesday 9th June 2009 04:24 GMT
And THE Bleedin' Obvious?! Diamond Geyser Rule, AC, which Personally Guarantees XXXXStreaming Fortunes to iCanny Personnel ...... Joint Venturing Virtual AIMachines into More Sticky Sweet Candy. ....... which would be in Quantum Communications Field akin to New Fangled Entangled Honey Traps/Blooming Flower Powers.
And as for the Virgin CLOUD and ITs Phorming Cloud Bases in CyberSpace, which you will not be surprised to learn Cloak Covers and Host SMARTer AIgents, which can Easily Zero in on Any and All Intelligence Led Operations, to lay Waste and/or overwhelm Systems with the Simple Disclosure of a Falsely Leading Truth which is always a Fatal Systemic Endemic Human Flaw stupidly carried forward into Binary Code in a Vain Bid to maintain a Previous Ineqitable and Positively Discriminatory Analogue Advantage.
However, Please be Cordially Advised, such is Considered and Deemed a Conscious Abusive Act in Virtualisation and Punitive Self Destructive Sanction Automatically Ensues to Purge the Systems of their Failings Preventing Future Travel ..... Magical Mystery Turing ....... and Virtual TelePortation Comunications Control of Global Events.
The SMARTer Operating System will Programme Accordingly to make Full and Beta Use of ITs Novel and Noble Facility/Faculties and all Others will Fail Miserably in the New SurReal Applied IntelAIgents Environment.
Which makes the Future Choice and Path to be Followed something of a No Brainer.
@AC 23:49 GMT
"In fact, you failed to give them reasonable time to respond before calling up the hackers" WHAT?!
Vendor has been given 2 weeks to show interest and provide any sort of update. Which he failed to do. You seem to think that vendor was asked to fix the bug in 2 weeks time, but there is nothing to support this.
Vendor was provided with the private link to bugreport, didn't access it, didn't provide any information when this would be fixed, in short he had shown no interrest in fixing the bugs. Now, what would YOU do, mr anonymous smarpants?
"Vendor has been given 2 weeks to show interest and provide any sort of update. Which he failed to do. You seem to think that vendor was asked to fix the bug in 2 weeks time, but there is nothing to support this."
On the contrary, in the section of the "security advisory" that I quoted it is clear that the vendor was replying to correspondence, but hadn't got around to dealing with it yet. Only a teenager without business experience (who else has the time to find bugs in other people's software for free?) would assume that this shows "no interrest [sic] in fixing the bugs". It's clear that Kloxo are lackadaisical about security, and I am in no way attempting to exculpate them -- indeed, looking at the vulns being exploited, they're complete t***ers -- but the fact they have problems is hardly unique in the IT industry, is it? That fact doesn't justify releasing these vulns so soon, and without warning. I can understand wanting a bit of kudos for finding all those bugs, but seriously...
You ask me what I would do. I would give the company a bit longer to respond, whilst embarrassing them with a public but non-specific security alert. If it took them more than a few months (let's say 6), *then* I would think about publication, and to hell with them. I would wait more than *3 days* for a follow-up to the last piece of correspondence...
@ Kev K
A backup on a second raid disk is great in all intents and purposes but I'd say for that extra bit of security a backup on a completely different machine at a different location maybe even with a different provider would be a good thing.
I keep occasionally badgering one of my customers I support about their backups. They have a server with RAID disks which they backup but the backup never goes off site because they only have one backup drive. No matter how much I tell them that they need AT LEAST one other backup drive so they can alternate the drives and have an off-site backup it just falls on deaf ears because they won't spend the money on a bog standard USB hard drive (they don't have much data, easily under 160GB!).
"Even if the service is unmanaged do they not run backups as standard incase of hardware failure ?? I would be a bit miffed if I was a customer there."
If they offer an specific package which does not include backup but comes at a lower price point, then you'd have 0 right to be 'miffed' if there weren't backups.
Such a package is just offering customers what they want: cost savings based on the customer sorting their own backup solution out.
Owner hangs himself - coincidence?
The report says the owner of lxlabs, has committed suicide