Hmmmm..

http://forum.lxlabs.com/index.php?t=msg&th=12365&start=0&

Seems like this has been known about for a bit but they don't seem to be doing anything about it!

Christ on a bike

That's not nice.

Even if the service is unmanaged do they not run backups as standard incase of hardware failure ?? I would be a bit miffed if I was a customer there.

I run a vps (cpanel) for some smaller clients & testing think I'm going to check my backups (2nd raid disk) and pull a set down to my local servers "just in case"

There really are some evil people with no fathers out there who probably also have carnal relationships on the maternal side

I was affected by this hack. I am still awaiting a response.

A server which was hosting a very large campaign website was one of the servers hosted and attacked. I am still waiting on VASERV to issue an update regarding my node... although so far they have dealt with the matter well.

What has happened is a serious criminal act and those involved should be brought to justice. I am surprised this has not been mentioned in the mainstream news.

This is a major wake up call.

So Pissed off

We did have backups of the majority of the sites - but so many have been lost / backups were out of date. VAServ should be ashamed that this happened. I was one week away from moving everything to VPS.net aswell.

Real hypervisors

they cost good money for a reason. Hopefully they'll learn to secure their servers better next time... Or at least others using this platform. I am sure this company is going to get a _huge_ flood of customers after this....

Lack of information from VAServe

I have an unmanaged VPS that has disappeared. I feel sorry for VAServe, other than choosing poor management software, they couldnt have predicted this! The VAServe status page is a bit sketchy, and i dont know which physical server my node was hosted on, so i have no idea if all my data has been lost, or when it might be restored :-(

Not zero-day at all?!

If find the use of the term "zero-day exploit" a bit rich as(according to http://www.milw0rm.com/exploits/8880) the vendors were notified on the 21st of may and the exploits has been 'in the wild' for a few days now. Strangely enough I have a VPS from these guys and it has not been affected, perhaps to do with my server being xen based....

Absurd . . .

"Low-end" customers get no backup service? Since it's a VM-based structure, surely one day a week, the "low-end" accounts could be taken offline, and the directory holding each VM's files be ZIPped or RARed?

I used to run some shell boxes in VM for people, and would, upon request, backup the contents for them - even though they paid me NOTHING for the service, nor for the backup procedure. Backups were password-protected and placed in a private FTP directory for the user to "collect" within a set period of time. if the user chose not to download their backups, they were copied to an external HD, as well as burned to DVDrom for safekeeping. I once actually snailmailed a user's backups to him, as he only had a dialup connection and as such wasn't up for the large download from FTP.

Mind you, my service was a hobby, and was only accessible to a select few "trusted" friends. It hosted no "sensitive" data. It was not "important", nor was it ever advertised.

Tux, because... WAAARK! (or whatever noise Penguins make).

Rubbish Software

if you look at the exploits on milw0rm, its laughable that lxlabs have even managed to string a product together. Im not a web apps expert but even i couldnt put out such shockingly bad software

No kidding ?

I think we'll have to wait a bit for the UK Glovinmint to wake up. Possibly until the Cabinet return from their jollies, after the summer recess.

ALF

Re: Real Hypervisors

@ Christopher Ahrens

HyperVM is not the hypervisor, it is a web based management tool for Xen and Virtuozzo which are both "real" virtualisation technologies.

The real culprit here is LX Labs for not delivering a secure product, despite their claims on their website:

http://lxlabs.com/software/kloxo/security/

"We take security as the most serious of the concerns and have worked hard to create a secure environment where you can be confident about the server's state."

This is clearly rubbish.

Ahhhhhh

eDarwin in action. The web could do with a bit less shite on it.

Now I know why

We haven't virtualized anything important in our environment.... poof ! So guys, where are those virtual backups ?

Not really a surprise

I'm still waiting to find out if my VPS was one that got rm'd, but if I'm honest I half suspected (along with some others here) that this might happen one day with HyperVM. Not because I knew anything about HyperVM itself, but because web apps in general pose so many security problems. There are usually many different input methods, all filtered differently and usually all with access to the crown jewels.

The annoying thing here is that if my VPS comes back on line do I assume the hacker(s) left a back-door somewhere, and rebuild it from scratch just to be safe? How very tiresome.

Paris because I saw this coming, ahem.

oOOPs??

oUCH. Cloud Computing has a couple embedded CumuloNimbi. Kind of bad when a management tool can enable so much virtual destruction. Where's the shadow copy service that allows you to take hot backups for this? At least stuff would be a little out of date, but not totally gone.

VAServ - very good

I'm a customer and I'm very happy with VAServ - they are a budget VPS provider who offer a no frills service which I use for personal stuff / development. The poster above who claims to be hosting a major campaign with them is clearly an idiot.

We use(d) HyperVM...

We're a UK and USA based provider who have had to cease all new orders and remove HyperVM from all our servers.

After running several tests on development hardware, the issues still remain -- LXLabs have essentially packed up and gone home.

Bill gates, because he loves hackers.

Backruptcy

I am a small web host, that had primary services hosted with VAServ and I expect that if I am not on-line within another 24 hours, I will have to dissolve my company.

I also know another person who's entire hosting infrastructure has been compromised and taken down by this attack.

I am surprised that they did not mirror data on their servers, and do understand that the challenge they face now to restore that many websites and servers. Is indeed no small job.

Did he piss off the wrong clients?

Vaserv subsidiary a2b2.net has something of a history of providing IP space for phishes and spam/phish maildrops, and as recently as last week was running a site offering turnkey phish kits from the same IP address that had recently hosted several bank and PayPal phishes:

http://tacit.livejournal.com/297618.html

http://tacit.livejournal.com/297775.html

http://tacit.livejournal.com/299317.html

Wouldn't surprise me one bit to learn that the attack was perpetrated by a current or former customer, somehow.

Bleedin' Obvious?!

Why are so many people moaning about lost data and then no idea where their backups are? For flips sake I have cruddy little personal webpage on some free provider with photos of me doing stupid stuff, but hell's teeth even I keep a backup of it, so it can be resent! I know you pay for services, but quite often you will find that cut-rate comes at a price.

If you have to trust someone else, DON'T! As an idiot manager once said to me, in one of his many David Brent moments, "We never assume, we always make sure. Right?".

Umm... Leet Speak for ???

It's all in a name

FSCKVPS Summary

More disasters caused by Micro$oft...

M$ really should take the blame for the disasters they cause, with these zero-day bugs infesting their crapplications.

Oh? Sorry? This wasn't Windows-based applications?

Oh, right, I'll stop ranting then.

They should have used OSX. No security holes in that, no sirree!

But seriously, a total bummer to everyone who was on it....

Mirrored data...

<blockquote>I am surprised that they did not mirror data on their servers, and do understand that the challenge they face now to restore that many websites and servers. Is indeed no small job.</blockquote>

The neat thing about mirrored data is that valid data manipulation such as deletion of unwanted data gets backed up to your backup system.

Mirroring data is only valid for guarding against hardware failures that don't cause bad data to be stored and replicated. Unless there is versioned backup to offline storage, there really is no backup at all.

backup

If you don't have regular backups, then you should know you are rolling the dice. If it wasn't this, then any number of other possible events could have taken out your data.

Seems like a lot of people learn this the hard way by losing a lot of data at some point in time.

If you used a hosting service without a backup plan, and then didn't create your own backup plan, you really set yourself up for this kick in the teeth.

I feel a disturbance in the intertubes...

....ah, 100,000 sites screaming out from the void....

If your business is reliant on your site/software.

I have only 3 things to say:

Backup

Backup

Backup

@Backruptcy

I hate to break this to you, but it's sort of your fault for relying on one provider.

Even if you're a low-end provider (hosting sites on someone else's VPS setup is low-end IMHO) it's not hard to get at least one box somewhere else and regularly smuggle your data back and forth. Having all your boxes at one provider is stupid for all sorts of reasons what happens to your DNS, mail etc when the provider has down time, or goes bankrupt and gets their cables pulled?

@AC 23:49 GMT

"In fact, you failed to give them reasonable time to respond before calling up the hackers" WHAT?!

Vendor has been given 2 weeks to show interest and provide any sort of update. Which he failed to do. You seem to think that vendor was asked to fix the bug in 2 weeks time, but there is nothing to support this.

Vendor was provided with the private link to bugreport, didn't access it, didn't provide any information when this would be fixed, in short he had shown no interrest in fixing the bugs. Now, what would YOU do, mr anonymous smarpants?

@Bronek Kozicki

"Vendor has been given 2 weeks to show interest and provide any sort of update. Which he failed to do. You seem to think that vendor was asked to fix the bug in 2 weeks time, but there is nothing to support this."

On the contrary, in the section of the "security advisory" that I quoted it is clear that the vendor was replying to correspondence, but hadn't got around to dealing with it yet. Only a teenager without business experience (who else has the time to find bugs in other people's software for free?) would assume that this shows "no interrest [sic] in fixing the bugs". It's clear that Kloxo are lackadaisical about security, and I am in no way attempting to exculpate them -- indeed, looking at the vulns being exploited, they're complete t***ers -- but the fact they have problems is hardly unique in the IT industry, is it? That fact doesn't justify releasing these vulns so soon, and without warning. I can understand wanting a bit of kudos for finding all those bugs, but seriously...

You ask me what I would do. I would give the company a bit longer to respond, whilst embarrassing them with a public but non-specific security alert. If it took them more than a few months (let's say 6), *then* I would think about publication, and to hell with them. I would wait more than *3 days* for a follow-up to the last piece of correspondence...

@Kev

"Even if the service is unmanaged do they not run backups as standard incase of hardware failure ?? I would be a bit miffed if I was a customer there."

If they offer an specific package which does not include backup but comes at a lower price point, then you'd have 0 right to be 'miffed' if there weren't backups.

Such a package is just offering customers what they want: cost savings based on the customer sorting their own backup solution out.