back to article Webhost hack wipes out data for 100,000 sites

A large internet service provider said data for as many as 100,000 websites was destroyed by attackers who targeted a zero-day vulnerability in a widely-used virtualization application. Technicians at UK-based Vaserv.com were still scrambling to recover data on Monday evening UK time, more than 24 hours after unknown hackers …

COMMENTS

This topic is closed for new posts.

Page:

  1. Anonymous Coward
    Thumb Down

    Hmmmm..

    http://forum.lxlabs.com/index.php?t=msg&th=12365&start=0&

    Seems like this has been known about for a bit but they don't seem to be doing anything about it!

  2. Kev K
    Thumb Down

    Christ on a bike

    That's not nice.

    Even if the service is unmanaged do they not run backups as standard incase of hardware failure ?? I would be a bit miffed if I was a customer there.

    I run a vps (cpanel) for some smaller clients & testing think I'm going to check my backups (2nd raid disk) and pull a set down to my local servers "just in case"

    There really are some evil people with no fathers out there who probably also have carnal relationships on the maternal side

  3. R.E.H.

    Not surprised

    I have (had!) a backup server hosted by these folks... I used to have some more important stuff there, but pulled it out a few months ago because HyperVM was making me nervous. They pulled the entire control panel down several times since recently due to suspected vulnerabilities in the software.

    Basically, HyperVM looks like a house of cards so I think it was only a matter of time before it got hacked. The control panel appears to run as root on each VPS host, of course any outward-facing thing can get hacked but there ought to have been some level of abstraction between the control panel and the VPSes to slow down the hackers. Doesn't seem like there was though.

    Pretty glad I moved my stuff when I did.

  4. Anonymous Coward
    Alert

    I was affected by this hack. I am still awaiting a response.

    A server which was hosting a very large campaign website was one of the servers hosted and attacked. I am still waiting on VASERV to issue an update regarding my node... although so far they have dealt with the matter well.

    What has happened is a serious criminal act and those involved should be brought to justice. I am surprised this has not been mentioned in the mainstream news.

    This is a major wake up call.

  5. Daniel Voyce
    Thumb Down

    So Pissed off

    We did have backups of the majority of the sites - but so many have been lost / backups were out of date. VAServ should be ashamed that this happened. I was one week away from moving everything to VPS.net aswell.

  6. Christopher Ahrens
    Alert

    Real hypervisors

    they cost good money for a reason. Hopefully they'll learn to secure their servers better next time... Or at least others using this platform. I am sure this company is going to get a _huge_ flood of customers after this....

  7. Ian Nice
    Thumb Down

    Lack of information from VAServe

    I have an unmanaged VPS that has disappeared. I feel sorry for VAServe, other than choosing poor management software, they couldnt have predicted this! The VAServe status page is a bit sketchy, and i dont know which physical server my node was hosted on, so i have no idea if all my data has been lost, or when it might be restored :-(

  8. will
    Thumb Down

    Not zero-day at all?!

    If find the use of the term "zero-day exploit" a bit rich as(according to http://www.milw0rm.com/exploits/8880) the vendors were notified on the 21st of may and the exploits has been 'in the wild' for a few days now. Strangely enough I have a VPS from these guys and it has not been affected, perhaps to do with my server being xen based....

  9. Ceiling Cat
    Linux

    Absurd . . .

    "Low-end" customers get no backup service? Since it's a VM-based structure, surely one day a week, the "low-end" accounts could be taken offline, and the directory holding each VM's files be ZIPped or RARed?

    I used to run some shell boxes in VM for people, and would, upon request, backup the contents for them - even though they paid me NOTHING for the service, nor for the backup procedure. Backups were password-protected and placed in a private FTP directory for the user to "collect" within a set period of time. if the user chose not to download their backups, they were copied to an external HD, as well as burned to DVDrom for safekeeping. I once actually snailmailed a user's backups to him, as he only had a dialup connection and as such wasn't up for the large download from FTP.

    Mind you, my service was a hobby, and was only accessible to a select few "trusted" friends. It hosted no "sensitive" data. It was not "important", nor was it ever advertised.

    Tux, because... WAAARK! (or whatever noise Penguins make).

  10. This post has been deleted by its author

  11. Anonymous Coward
    Anonymous Coward

    Wrong.

    @Christopher Ahrens: They DID secure their servers, LXLABS (and their crap software) had a exploit which basically allowed root access to anyone running HyperVM.

    You can find information here: http://www.webhostingtalk.com/showthread.php?t=867100

  12. Gav

    Status

    I don't think this is the fault of Vaserv, their software was fully up to date - I don't see why they should be ashamed as they take security very seriously.

    Their status site is here for anyone that needs it http://vaserv.com/

  13. Tim
    Thumb Down

    Rubbish Software

    if you look at the exploits on milw0rm, its laughable that lxlabs have even managed to string a product together. Im not a web apps expert but even i couldnt put out such shockingly bad software

  14. Anonymous Coward
    Anonymous Coward

    You get what you pay for

    More info here:

    http://www.webhostingtalk.com/showthread.php?t=867100

    and here:

    http://66.71.245.2/~vaservc/

    I host a number of VPS's with vaserve, their communication has been pretty good through this. They are a no frills host and it's pretty clear that if you want backups, you need to make sure you get backups.

  15. Andrew Fraser

    Now replace web services,

    With cloud managed servers, and you have another reason cloud is a bad idea (tm)

  16. Alfazed
    Unhappy

    No kidding ?

    I think we'll have to wait a bit for the UK Glovinmint to wake up. Possibly until the Cabinet return from their jollies, after the summer recess.

    ALF

  17. Automatic jack
    Stop

    Re: Real Hypervisors

    @ Christopher Ahrens

    HyperVM is not the hypervisor, it is a web based management tool for Xen and Virtuozzo which are both "real" virtualisation technologies.

    The real culprit here is LX Labs for not delivering a secure product, despite their claims on their website:

    http://lxlabs.com/software/kloxo/security/

    "We take security as the most serious of the concerns and have worked hard to create a secure environment where you can be confident about the server's state."

    This is clearly rubbish.

  18. Sarev
    Flame

    Ahhhhhh

    eDarwin in action. The web could do with a bit less shite on it.

  19. Anonymous Coward
    Alert

    Now I know why

    We haven't virtualized anything important in our environment.... poof ! So guys, where are those virtual backups ?

  20. Anonymous Coward
    Anonymous Coward

    I've had swift responses but it shouldn't have happened.

    Of my two VPSs one has been unreachable for over 24hrs. :(

    Over the last year I've had a good experience with Vaserv and consider them quite low on bullshit. Even today I've emailed them twice and had replies in 2 minutes and 15 minutes. That is VERY fast given the scale of the problem they are facing. Obviously this whole situation shouldn't have happened and I'm facing data loss. However I could have bought the add on back up from them, used the web based control panel to do it or used rsync. So I can't really blame them for the lack of a more recent backup.

    Initially they did a good job of keeping the information flowing. But then they listed some damaged nodes and then claimed "Everything else should be up and running for the UK". This was a mistake because it seems to be inaccurate, it raised expectations and I imagine it flooded them with queries about why things weren't working. Tired people make mistakes. Anyway hope this resolves as quickly as possible for all our sakes. I'm not very happy so my staying with them depends on their response to this exceptional disaster.

  21. Anonymous Coward
    Paris Hilton

    Not really a surprise

    I'm still waiting to find out if my VPS was one that got rm'd, but if I'm honest I half suspected (along with some others here) that this might happen one day with HyperVM. Not because I knew anything about HyperVM itself, but because web apps in general pose so many security problems. There are usually many different input methods, all filtered differently and usually all with access to the crown jewels.

    The annoying thing here is that if my VPS comes back on line do I assume the hacker(s) left a back-door somewhere, and rebuild it from scratch just to be safe? How very tiresome.

    Paris because I saw this coming, ahem.

  22. gollux
    Alert

    oOOPs??

    oUCH. Cloud Computing has a couple embedded CumuloNimbi. Kind of bad when a management tool can enable so much virtual destruction. Where's the shadow copy service that allows you to take hot backups for this? At least stuff would be a little out of date, but not totally gone.

  23. Anonymous Coward
    Anonymous Coward

    @Jonathan Zahedieh

    Jonathan,

    You wrote:

    "The researcher than found the vulnerabilities in the only gave the developer 2/3 weeks to fix it, looks like to me that he got peed off and released them to the public because lack of response."

    I'd like to clarify this. First, I did not give the developer "2/3 weeks to fix" the issues. In fact, I did not give any timeline at all. What I did is what the advisory says. Had the vendor looked at the issues (which was not done) and requested some time to address them, of course, I would have given any amount of time requested before going public with the information.

    Second, what it looks like to you and what really happened are 2 different things. I did not release the information out of anger with anyone. It was released so that customers, both current and potential, would be aware of the issues. It is not the job of the person who spends their time finding and documenting the bugs to babysit a vendor, plain and simple.

    Third, as it stands, there is nothing whatsoever that definitively connects the current situation with the afflicted webhost with the information that was made publicly available. I audited Kloxo. As I understand it, and do correct me if I'm wrong, but they believe the issue was with HyperVM. I did not find out until later than HyperVM and Kloxo (formerly LxAdmin) share some of the same features/code.

    Finally, lxlabs/kloxo/hypervm has been getting hacked for a while now, well before I ever published anything. Read their forums and you will see.

  24. Anonymous Coward
    Anonymous Coward

    Boyaka Shaka

    they must have pissed off the wrong person. Hacks of these styles tend to be quite personal.

  25. Will Jenkins
    Happy

    VAServ - very good

    I'm a customer and I'm very happy with VAServ - they are a budget VPS provider who offer a no frills service which I use for personal stuff / development. The poster above who claims to be hosting a major campaign with them is clearly an idiot.

  26. Anonymous Coward
    Anonymous Coward

    Self-interest

    In reply to the above, here is an excerpt from that advisory:

    # Timeline :

    # 05/21/2009 - sent initial email to vendor with a link to a private resource for viewing various kloxo hiab575 vulnerability info

    # 05/23/2009 - received the following: "Thanks for the info. I will review this and let you know." (no signature)

    # 05/30/2009 - sent an email asking if there were any updates

    # 06/01/2009 - received the following: "Sorry for the delay. I am currently looking into this, and will reply in a couple of hours time." (no signature)

    # 06/04/2009 - nothing heard from vendor, and the private resource containing the vulnerability info still does not appear to have been accessed

    # 2 weeks have passed since the initial notification. Vendor appears uninterested.

    Your qualification to call yourself a white hat is based on the pretence of being interested in giving Kloxo information about weaknesses that they could fix privately. In fact, you failed to give them reasonable time to respond before calling up the hackers. There's no defence.

  27. This post has been deleted by its author

  28. Rick Mills
    Gates Horns

    We use(d) HyperVM...

    We're a UK and USA based provider who have had to cease all new orders and remove HyperVM from all our servers.

    After running several tests on development hardware, the issues still remain -- LXLabs have essentially packed up and gone home.

    Bill gates, because he loves hackers.

  29. Anonymous Coward
    Anonymous Coward

    Fig leaf

    "you gave people the warning to make a back up"

    Where? He didn't even warn the company of what he was about to do. That would have woken them up.

    The "full disclosure" cant is just a fig leaf in this case. I agree he can't be expected to babysit these stupid lazy tw*ts at Kloxo, but at the same time he overstepped the mark by a long way in this case. Knowledge gives power and privilege, but where was the _noblesse oblige_? He should learn from this, unless he just wants to create problems for himself in the future, especially given the dubious legal situation.

  30. Anonymous Coward
    Linux

    Backruptcy

    I am a small web host, that had primary services hosted with VAServ and I expect that if I am not on-line within another 24 hours, I will have to dissolve my company.

    I also know another person who's entire hosting infrastructure has been compromised and taken down by this attack.

    I am surprised that they did not mirror data on their servers, and do understand that the challenge they face now to restore that many websites and servers. Is indeed no small job.

  31. Franklin
    Black Helicopters

    Did he piss off the wrong clients?

    Vaserv subsidiary a2b2.net has something of a history of providing IP space for phishes and spam/phish maildrops, and as recently as last week was running a site offering turnkey phish kits from the same IP address that had recently hosted several bank and PayPal phishes:

    http://tacit.livejournal.com/297618.html

    http://tacit.livejournal.com/297775.html

    http://tacit.livejournal.com/299317.html

    Wouldn't surprise me one bit to learn that the attack was perpetrated by a current or former customer, somehow.

  32. Anonymous Coward
    Flame

    Bleedin' Obvious?!

    Why are so many people moaning about lost data and then no idea where their backups are? For flips sake I have cruddy little personal webpage on some free provider with photos of me doing stupid stuff, but hell's teeth even I keep a backup of it, so it can be resent! I know you pay for services, but quite often you will find that cut-rate comes at a price.

    If you have to trust someone else, DON'T! As an idiot manager once said to me, in one of his many David Brent moments, "We never assume, we always make sure. Right?".

  33. gollux
    Happy

    Umm... Leet Speak for ???

    It's all in a name

    FSCKVPS Summary

  34. Adrian Esdaile
    Joke

    More disasters caused by Micro$oft...

    M$ really should take the blame for the disasters they cause, with these zero-day bugs infesting their crapplications.

    Oh? Sorry? This wasn't Windows-based applications?

    Oh, right, I'll stop ranting then.

    They should have used OSX. No security holes in that, no sirree!

    But seriously, a total bummer to everyone who was on it....

  35. gollux
    Alert

    Mirrored data...

    <blockquote>I am surprised that they did not mirror data on their servers, and do understand that the challenge they face now to restore that many websites and servers. Is indeed no small job.</blockquote>

    The neat thing about mirrored data is that valid data manipulation such as deletion of unwanted data gets backed up to your backup system.

    Mirroring data is only valid for guarding against hardware failures that don't cause bad data to be stored and replicated. Unless there is versioned backup to offline storage, there really is no backup at all.

  36. Anonymous Coward
    Boffin

    backup

    If you don't have regular backups, then you should know you are rolling the dice. If it wasn't this, then any number of other possible events could have taken out your data.

    Seems like a lot of people learn this the hard way by losing a lot of data at some point in time.

    If you used a hosting service without a backup plan, and then didn't create your own backup plan, you really set yourself up for this kick in the teeth.

  37. Winkypop Silver badge
    Alien

    I feel a disturbance in the intertubes...

    ....ah, 100,000 sites screaming out from the void....

    If your business is reliant on your site/software.

    I have only 3 things to say:

    Backup

    Backup

    Backup

  38. Mark y

    @Backruptcy

    "I am a small web host, that had primary services hosted with VAServ and I expect that if I am not on-line within another 24 hours, I will have to dissolve my company.

    I also know another person who's entire hosting infrastructure has been compromised and taken down by this attack.

    I am surprised that they did not mirror data on their servers, and do understand that the challenge they face now to restore that many websites and servers. Is indeed no small job."

    If your time line is only 24 hours from when you posted I take it you made sure your SLA with them for restoration of service was well within this time-frame when you signed up?

    I doubt you did and I can have very little sympathy for someone who appears to be sent under by this because they failed to address the issue of tail events and business continuity.

  39. Daniel Palmer
    Flame

    @Backruptcy

    I hate to break this to you, but it's sort of your fault for relying on one provider.

    Even if you're a low-end provider (hosting sites on someone else's VPS setup is low-end IMHO) it's not hard to get at least one box somewhere else and regularly smuggle your data back and forth. Having all your boxes at one provider is stupid for all sorts of reasons what happens to your DNS, mail etc when the provider has down time, or goes bankrupt and gets their cables pulled?

  40. amanfromMars Silver badge

    The CLOUD is Open and Virtually Closed for Banking Business.

    "If you have to trust someone else, DON'T! As an idiot manager once said to me, in one of his many David Brent moments, "We never assume, we always make sure. Right?"." ..... By Anonymous Coward Posted Tuesday 9th June 2009 04:24 GMT

    And THE Bleedin' Obvious?! Diamond Geyser Rule, AC, which Personally Guarantees XXXXStreaming Fortunes to iCanny Personnel ...... Joint Venturing Virtual AIMachines into More Sticky Sweet Candy. ....... which would be in Quantum Communications Field akin to New Fangled Entangled Honey Traps/Blooming Flower Powers.

    And as for the Virgin CLOUD and ITs Phorming Cloud Bases in CyberSpace, which you will not be surprised to learn Cloak Covers and Host SMARTer AIgents, which can Easily Zero in on Any and All Intelligence Led Operations, to lay Waste and/or overwhelm Systems with the Simple Disclosure of a Falsely Leading Truth which is always a Fatal Systemic Endemic Human Flaw stupidly carried forward into Binary Code in a Vain Bid to maintain a Previous Ineqitable and Positively Discriminatory Analogue Advantage.

    However, Please be Cordially Advised, such is Considered and Deemed a Conscious Abusive Act in Virtualisation and Punitive Self Destructive Sanction Automatically Ensues to Purge the Systems of their Failings Preventing Future Travel ..... Magical Mystery Turing ....... and Virtual TelePortation Comunications Control of Global Events.

    The SMARTer Operating System will Programme Accordingly to make Full and Beta Use of ITs Novel and Noble Facility/Faculties and all Others will Fail Miserably in the New SurReal Applied IntelAIgents Environment.

    Which makes the Future Choice and Path to be Followed something of a No Brainer.

  41. Bronek Kozicki
    Flame

    @AC 23:49 GMT

    "In fact, you failed to give them reasonable time to respond before calling up the hackers" WHAT?!

    Vendor has been given 2 weeks to show interest and provide any sort of update. Which he failed to do. You seem to think that vendor was asked to fix the bug in 2 weeks time, but there is nothing to support this.

    Vendor was provided with the private link to bugreport, didn't access it, didn't provide any information when this would be fixed, in short he had shown no interrest in fixing the bugs. Now, what would YOU do, mr anonymous smarpants?

  42. Ian Rons
    IT Angle

    @Bronek Kozicki

    "Vendor has been given 2 weeks to show interest and provide any sort of update. Which he failed to do. You seem to think that vendor was asked to fix the bug in 2 weeks time, but there is nothing to support this."

    On the contrary, in the section of the "security advisory" that I quoted it is clear that the vendor was replying to correspondence, but hadn't got around to dealing with it yet. Only a teenager without business experience (who else has the time to find bugs in other people's software for free?) would assume that this shows "no interrest [sic] in fixing the bugs". It's clear that Kloxo are lackadaisical about security, and I am in no way attempting to exculpate them -- indeed, looking at the vulns being exploited, they're complete t***ers -- but the fact they have problems is hardly unique in the IT industry, is it? That fact doesn't justify releasing these vulns so soon, and without warning. I can understand wanting a bit of kudos for finding all those bugs, but seriously...

    You ask me what I would do. I would give the company a bit longer to respond, whilst embarrassing them with a public but non-specific security alert. If it took them more than a few months (let's say 6), *then* I would think about publication, and to hell with them. I would wait more than *3 days* for a follow-up to the last piece of correspondence...

  43. Rob Beard

    @ Kev K

    A backup on a second raid disk is great in all intents and purposes but I'd say for that extra bit of security a backup on a completely different machine at a different location maybe even with a different provider would be a good thing.

    I keep occasionally badgering one of my customers I support about their backups. They have a server with RAID disks which they backup but the backup never goes off site because they only have one backup drive. No matter how much I tell them that they need AT LEAST one other backup drive so they can alternate the drives and have an off-site backup it just falls on deaf ears because they won't spend the money on a bog standard USB hard drive (they don't have much data, easily under 160GB!).

    Rob

  44. Ryan Barrett
    Go

    @Kev

    "Even if the service is unmanaged do they not run backups as standard incase of hardware failure ?? I would be a bit miffed if I was a customer there."

    If they offer an specific package which does not include backup but comes at a lower price point, then you'd have 0 right to be 'miffed' if there weren't backups.

    Such a package is just offering customers what they want: cost savings based on the customer sorting their own backup solution out.

  45. Anonymous Coward
    Anonymous Coward

    Owner hangs himself - coincidence?

    The report says the owner of lxlabs, has committed suicide

    http://timesofindia.indiatimes.com/Bangalore/Techie-hangs-himself-in-HSR-Layout-/articleshow/4633101.cms

  46. Alistair Macdonald
    Unhappy

    LXLabs Owner Hangs Himself!

    It appears this was all too much for the poor guy and the India Times is reporting that he hung himself yesterday :(

    http://timesofindia.indiatimes.com/Bangalore/Techie-hangs-himself-in-HSR-Layout-/articleshow/4633101.cms

    puts it all into perspective! This is what happens when a large chunk of the hosting industry base their entire businesses on a chap's "India-style-super-duper" software....

  47. MinionZero
    Unhappy

    This seems like a perfect storm of mistakes!

    To start with, backups are essential for any web site (in fact anything on a computer), not simply an added bonus feature as Vaserv wishes to treat it. That's a fundamental mistake in their management and now they are paying for their false economy of not properly backing up. Everyone involved should have backups of their own sites as well as Vaserv having mandatory backups. (I always keep my own backups of the entire site for peace of mind as well. That way even if the host burns down I still have my data).

    But that said, they have had the misfortune of suffering a shockingly bad hacker with an attitude of pure contempt. The hacker behind this attack is going to do serious time for this. It has to be one of the most expensive criminal hacks of all time?! ... 100000 web sites destroyed! ... the cost of destroying that many is serious money. How many thousands of businesses have been hit?! ... The lawyers are probably lining up around the building wanting to get in on all the ways cases and claims can be generated from so much destruction!

    I don't blame LXLabs for being unavailable to most people. Their phones must be running continuously. I feel sorry for the programmers as they are now in the center of a horrific storm.

    I don't think the full scale of this criminal action has sunk in yet. The major news channels and papers should be picking up on this.

  48. Matthew Macdonald-Wallace
    Flame

    Me Too!

    A number of sites hosted by me for friends and family disappeared as a result of this hack, including a site for a business that I was going to set up last year.

    I have a complete backup because it was obvious from the site that backups and data security was not something that they provided.

    If you lost data in this, then I'm sorry but you should have read the T's & C's and taken appropriate steps.

    Moral: Don't host business critical services on a system that costs less than £20 per month...

  49. Matthew Macdonald-Wallace
    Thumb Up

    Further to the above...

    I've just had an email from vaserv who have advised me that:

    ================

    Currently, we are enrolling a new platform on the new hardware for our customers who have lost all their data on one of the, unfortunately, lost host machines, and the ones that do have backups and would like to get things as soon as possible. Currently we expect to start deploying these during the night, once servers are prepared and installed.

    We would also take this opportunity to outline that we will be issuing full RFO (Reason For Outage) and some other announcements related to this situation once everything is fully operative again. In addition to this, please read what we will be doing for you, our beloved customers, below:

    - We will be applying 1 month worth of credits in case you have had a downtime for the day

    - We will be applying 2 months worth of credits in case you have suffered lost of your data

    ========================

    They've done a great job on this one, and I've even had an email from BlueSquare advising me that they will transfer VPS server to their own infrastructure ASAP, WITHOUT changing the pricing!

    Well done VAServ,

    A Very Happy Customer.

  50. Anonymous Coward
    Unhappy

    @MinionZero

    "I don't blame LXLabs for being unavailable to most people. Their phones must be running continuously. I feel sorry for the programmers as they are now in the center of a horrific storm."

    There was no "they" - only a "he" - which you think might have sounded a few warning bells to large companies.

    Now that he's dead and the vuln won't be fixed, I suspect there will be a few IT managers with some explaining to do.

Page:

This topic is closed for new posts.

Other stories you might like