We have a (unofficial) chat network setup to help clients, it is also open to discussion about anything that happened:
The director of an internet service provider has denied public allegations that poor password management and server configurations were responsible for an attack that wiped out data for more than 100,000 websites. Rus Foster, director of VAServ.com, also says he was shocked when he learned the head of an Indian software firm …
We have a (unofficial) chat network setup to help clients, it is also open to discussion about anything that happened:
Very sad to hear of this. I did business with Rus since his early days, still am a customer as it goes. I must be one of the lucky ones - my VPS is up and it still has all its data. I noted the LXadmin issues many months ago when there was another security issue raised about the LxAdmin product, its on the WHT forums somewhere.
Good luck with everything Rus, Im still sticking with Vaserv
Because sooner or later, catastrophic data loss will happen to you.
And please don't call the b@stards 'hackers'.
Script-kiddie or cracker is more like it.
I've been hosting with A2B2 for about 3 years now, after a succession of previous poor hosts. During this time I've had a competent and professional service. I think it's a real shame that Rus is losing his business over this.
the concept is flawed anyhow. Much better to have a dedicated server. The Visor will add to the overhead of the total system anyhow, it is not green to do this.
And, it is not fully tested, these people are guinea pigs to this setup, and don't go on about mainframes, the ZX Spectrum was outperforming those.
Simplistic and IT have never worked together, much better to keep it separate and complex and apply the concurrency and distribution at another level. There is a reason to move most stuff into UserSpace it keeps the kernel focussed, trying to distribute the kernel is a bit daft, it is tied to the hardware.
So, this is all about the adage don't use the same thing across many systems, and this what distributed virtualisation is up against, trying to make one across what is many.
Don't they get Star Trek in India, this is the flaw of the borg; lose of individuality, interoperable state machines that keep a degree of individuality is what is looking like the real winner. And that setup is far more like Star Fleet.
Right off to get my Tasha Yar, Diana Troy and One of Seven nodes interoperating in some sort of node fest, the Ryker node is down (no one cares) and every node has their phasers set to stun.
Honestly, how can some people be so thick as to feel there is any merit in wrecking somebody else’s systems. The only point they are proving is that they can destroy things. Same as a bunch of teenagers steeling a car, going for a joy-ride, crashing the car and then setting it on fire. Same level of intellect and reason required for either activity. If anyone claims credit for having done it then they are a target for the authourities. So it does not help the people who do it, does not help the people being done by it... so where's the good in this? Idiots!
...and no, not a bitter customer, just adding my opinion!
<< "Z3r0 day in hypervm??" the anonymous poster wrote, substituting numbers for letters as is common in hacker parlance. "Plz u give us too much credit." >>
It looks like a skiddy, it talks like a skiddy... I think there's a possibility we *are* giving them too much credit.
"the concept is flawed anyhow. Much better to have a dedicated server." --- WHAT?
I don't need (or want to pay for) a 16 core dedicated server to run a little proxy (to get round BBC's iplayer restrictions) and some tools for work (external host to check defenses etc.) I run a couple of scripts. So you would rather we have 100 individual servers running at 1% utilized using all that power and generating heat rather than 2 boxes running 100 virtual servers?
I have been with vaserv for a while now and they offer exactly what I want - cheap, small linux box.
...you needs it! And perhaps you should let your engineers practise with live data to test servers restorations once in a while.
No servers are invulnerable. It only takes one 0-day exploit or a slow patch to destroy years of work.
Daily backups. Learn it. Use it. Test it.
Definitely sounds like a skiddy.
After all, what hacker breaks into a system and then deletes everything? Any idiot can do a format.
Trouble is you get spotted very quickly and don't actually get anything from it. Far better to go deep, go silent and see what useful data you can steal. Credit card details, email addresses etc.
The only time I would expect to see a hacker nuke servers like that would be as part of a blackmail attempt.
Yes, the people that did this are idiots. Clearly what they lack are social skills. They're probably children sat in their bedrooms on the computer all night instead of out playing with friends, or socialising. I'd much rather spend my evenings out tinkering with the ladies than in tinkering with servers! I can't see what they get out of this sort of thing.
Paris, for all the obvious tinkering puns
the easiest target is dead so now they search somewhere else ......
As I understand it, a lot of host services of this type can keep costs low simply because they don't run backups which obviously involve time, effort and cost.
Its only fair though that hosting services specifically inform the prospective customer when he/she signs up, and then its up to them whether they want to be sensible or irresponsible. I can't understand why some people find this concept so difficult to understand!
They should have backed up everything because backups are like building insurance. Its an extra unwelcome expense but its needed to help survive catastrophic events. What they have suffered is effectively like a virtual form of arson totally destroying their business infrastructure.
But that said, its very sad to see the people caught up in this case being forced to suffer so much due to an act of outright ruthless destruction on a massive scale. I wonder how many small businesses are also suffering now they have had their web sites wiped out and have lost contact with their customers. This act of wanton destruction shows a ruthless intent by this hacker or hackers and they have to be caught. It never ceases to amaze me how a minority of people keep intentionally setting out to inflict suffering on others. They must be deeply troubled people themselves to intentionally want to cause others harm, but that doesn't excuse their behaviour. Bring others down for their entertainment shows what kind of people they really are.
The problem for all of us however isn't just these ruthless evil minded people. The even bigger problem is the result of their actions is going to force far greater controls to be forced on us all, because the authorities will be only to happy to use cases like this, as an excuse to clamp down and monitor the Internet far more closely and they have a valid point. We do need to catch and stop people like this. The problem is however the more power we give a government to clamp down and control people, the more they will use that power for their own gain as well, so its a double edged sword. Dammed if we do and dammed if we don't.
There is however an alternative if people don't want the centralised total government control solution then the only other solution I can see that would work is to educate everyone from school age upwards in basic psychology for their own protection to make sure everyone can see there are people in this world who intentionally set out to harm others and to see why they behave the way they do. Then and only then, would most people finally act to protect themselves and their businesses from the hostile intentions and actions of a minority of people who seek to harm others. Currently too many people in the world are too naïve and trusting simply because they don't understand the mentality of a minority who cause harm to others and sadly its cases like this that show there are people who set out to harm others.
The education route has never been tried on a global scale from school age upwards but it would totally change societies into becoming far more aware of the reasons behind this kind of behaviour and so societies would much more rapidly and strongly move to stop and block this kind of behaviour where ever it was found. The spread of knowledge throughout history has had the potential to help large numbers of people. All it would take is the will to work to spread it and the Internet is now able to spread knowledge far more than at any other point in history. The irony is the minority of people who set out to treat others with contempt so often don't want people to see why they behave the way they do, as it would stop them from getting away with their behaviour and reveal why they behave the way they do. So they intentionally say anything they can to hide their true reasons. Psychologists have learned to see through this behaviour. Its time everyone learns for their own protection.
We have a choice. The education route or the total government control route. Problem is freedom and the Internet only have one of these options. The control route is the end of the Internet as we know it.
...not the same as the gambling company is it?
I'm betting that the bad guys knew about this for a while. How can it be a zero-day if they already knew and it just took the good guys some time to find it?
Sometimes you don't realize the fence is broken until you see cows on the road...
I agree... The only reason a "hacker" would do this is part of a blackmail attempt... or to remove evidence of something already done.
200+ servers running how many virtual machines each? Sounds like a prime candidate for botnet control, stealth proxies, temporary harvesting depositories, bounce stations, or any number of useful setups, especially if the physical machine's norms are unusual or sporadic usage patterns.
What would you bet that this was an attempt to "clean up" after themselves once they were already done (or the way in was exposed), and what are odds for/against government sponsored tie-ins?
I have not run into a hosting company that does NOT tell you when you are getting automated backups. I also have not run into any hosting service that provides free backup services for dirt cheap accounts.
So if your provider does not EXPLICITLY state that you are getting backups, you must KNOW that you are responsible for that activity. It is unfortunate that VAserv had so many systems taken down, but if the machines are again available, the customers should get to work restoring from their own backups, and if they do not have any, then they should not be blaming VAserv for that. Frankly, it seems like VAserv is doing heroic work attempting to recover data that they were under no obligation to retain. Check out http://www.vaserv.com for status reports.
Even with our dedicated servers' nightly backups to tape maintained by the provider we STILL do our own nightly incremental backups via cronjob/rsync of all content files and databases. Even if the provider's data centers implode and we need to get completely new systems, we'll only be couple of hours away from full recovery, not counting propagation of the new DNS info.
In my last job at a hosting company, this is actually largely true. There were a lot of duplicate passwords, many of them having slight changes depending on the server, and a few key 'master' passwords.
When you have logins for the billing system, inventory/management, key authentication (for control panels etc.), KVM/IPs, and more, what are you going to do? Make obtusely different passwords for each section? And then do what, have a master password sheet you pass around? Nah, make it easy for the employees. It sounds insecure as hell, and it really is, and even as the most security-anal person there, I understood just why. Some of my coworkers used password vaults for easy access.
While I'm not familiar with how HyperVM manages file systems for virtualized containers, I know that in Virtuozzo it's very easy to fuck with a VPS from the hardware node itself -- just go into /vz/private and have fun. You can just nuke /vz and you've destroyed the files for every VPS in there except the config files (stored in /etc/vz/conf).
Billing system as well...most hosting companies, aside from small ones or immensely huge (ie, GoDaddy) ones use standardized billing platforms like WHMCS or billing software provided by their control panel provider (Plesk Billing, ClientExec, etc.). So as long as you know the database structure (and most use something similar), you can get in there and harvest away by dumping tables to an external file then using a few basic commands to make the output into an easily-readable file.
I still have no doubt that these are script kiddies, their behavior and actions speak volumes of it. But the attack itself sounds highly probable.