Feeds

back to article Code Spaces goes titsup FOREVER after attacker NUKES its Amazon-hosted data

Source code hosting provider Code Spaces has suffered the ultimate cloud nightmare, having been effectively forced out of business by the actions of an attacker who managed to gain access to its Amazon EC2 control panel. The devastating incident began on June 17 when Code Spaces – a company that claimed to offer "Rock Solid, …

COMMENTS

This topic is closed for new posts.

Page:

Complete Bastards

That is all.

10
0
Bronze badge

Complete incompetent bastards. No decent backups, no disaster recovery plan, no business continuity plan, nothing nada.

inexcusable for an IT business.

48
0
Silver badge

But a brilliant example of why one should NOT trust the cloud with strategic data.

They had their entire business in the cloud, and the cloud went away - with their business.

As a cautionary tale, it is strikingly effective.

61
2
Anonymous Coward

Re: "inexcusable for an IT business."

But not untypical of the foolish ones who try to convince us that "the cloud" can replace "the competent" and cost far less, shurely?

23
1

That's just utter bo**ocks. This has nothing to do with the cloud and everything to do with truly dreadful system administration. It could just as well have happened in a private data centre as on a cloud service. Cloud services may have their faults, but this is categorically not an example of one of them.

8
22
Bronze badge
FAIL

Trust the cloud?

In effect it looks like they trusted their whole business to a single public-facing login point?!

Fuck me, is this what 'the cloud' has done to layered security?

10
0
Bronze badge

This sort of thing can happen in a data centre, but that's a problem with outsourcing in general. It's usually done because outsourcing is cheaper, and it's usually done cheaper because either staff are outsourced to india and they pay them peanuts or you discover that the reason they can provide it cheaper than you can in house despite using the same suppliers is that your in house solution had redundant discs in RAID and backups, and theirs didn't.

2
0
Bronze badge
WTF?

@ Dr Who -

"Cloud services may have their faults, but this is categorically not an example of one of them."

Yeh, tell that to the users mate, I'm sure they will agree with you.

Vapourware and its data - one puff and it's gone. Wouldn't have happened with decent backups which aren't managed on-line. What a complete fail that is.

5
3

On the contrary, it is a good example. An internal data center wouldn't have had an easy to use WEB ACCESSIBLE front end you could use to cause all that damage.

All the crook needed was a single username/password to get in.

Plus, cloud pushes you to do everything on it, even when it's a bad idea. (See backups in the above article). This creates a single point of failure.

Sure, the administrators were idiots NOT to have backups elsewhere, but cloud helped push them in that position.

13
1
Bronze badge
Pint

"This has nothing to do with the cloud..."

It does in the sense that an actual old-school IT server room would likely have some actual backup tapes or portable HDDs locked up in a cupboard, or trucked away to a physical (not virtual) off-site backup. When EVERYTHING is in the cloud, then it is susceptible to hackers in the cloud. When your backups are physical, then the hackers have to be physically present (unless they log into your UPS and cause it to burn the building down, only hours before it would have done so anyway).

Corrective action and lesson learned is multi-layered, including having non-virtual backups that are also available on the Internet.

4
0
Bronze badge
FAIL

Re: "inexcusable for an IT business."

Note that in their explanation here: http://pastebin.com/WvtjMe9T

They state: Upon realisation that somebody had access to our control panel we started to investigate how access had been gained

It should, instead, have been: Upon realisation that somebody had access to our control panel we took down the website and disconnected all our systems from the internet..

Although that might be too much work for people working from home.

3
0
Bronze badge

The article sez that "offsite" backups were lost as well. WTF?

The offsite BUs were connected to the online servers?

The entire thing sounds like poor management...

5
0
Gold badge
Unhappy

@Peter 2

"Complete incompetent bastards. No decent backups, no disaster recovery plan, no business continuity plan, nothing nada.

inexcusable for an IT business."

They figured Amazon would handle everything.

They were wrong.

S**t happens. Guaranteed. It's not if you will attacked (as a successful business) it's when.

1
0

Re: "This has nothing to do with the cloud..."

"It does in the sense that an actual old-school IT server room would likely have some actual backup tapes or portable HDDs locked up in a cupboard, or trucked away to a physical (not virtual) off-site backup. When EVERYTHING is in the cloud, then it is susceptible to hackers in the cloud."

But it doesn't. Because EVERY business should have off site backups. It doesn't matter if you use the cloud, a datacenter, a colocation or your basement. Ate least ONE copy of the backups should be kept off site - and not accessible by the systems being backed up (ie: the backup can access the systems, to do a restore or a backup - but the systems can't access the backups).

Yes, the single sign on system that Amazon uses made it easier. No doubt about it. But the absence of off site backups... that was just neglect, not a cloud problem.

4
0

Re: "This has nothing to do with the cloud..."

Even if they had substantial and appropriate backups, there still would have been massive disruption to their business and their customers. Arguing that it has nothing to do with the cloud ignores this point.

I think the other important point is the lack of layered security - a single login portal is surely a major flaw, though I don't know if that is 'cloud typical' or not.

2
0
Bronze badge

Re: "This has nothing to do with the cloud..."

"Even if they had substantial and appropriate backups, there still would have been massive disruption to their business and their customers."

Which is why you have a business continuity plan, which is a (tested) plan as to how you are going to continue the business come what may.

2
0
Silver badge
Facepalm

Rather irresponsible

Code Spaces couldn't find $12.99 down the back of the sofa for a TFA key fob?

https://aws.amazon.com/iam/details/mfa/

39
1
JLV
Bronze badge

Re: Rather irresponsible

Or download a free tfa app to their mobile.

1
0
Mushroom

Bloody hell.

3
0
Silver badge
FAIL

Backups

It's not called backups if it's on the same system (or even same building), that is called a copy.

66
0
Anonymous Coward

Re: Backups

Yet every day I hear how the cloud and duplicating means the death of tape.....

36
2
Bronze badge

Re: Backups

The only death involved with tape is people who aren't using it. Then again, these sort of people are the sort of inept muppets who don't change their tapes, or just leave the same tape in the drive to be overwritten constantly, so if these people had have been using tape then it probably have been written off along with everything else.

Cynical, moi?

Ok, I might be mildly paranoid, but I still I find it comforting to know that I have everything needed to recover from the worst possible disaster imaginable sitting off site and offline.

21
1
JDX
Gold badge

Re: Backups

They had perfectly good backups - the backups didn't fail, someone with the authority to do so deleted them! They could've stored their backups in different locations across different servers but if you let someone get in and delete the backups, that is a process fail, not an infrastructure fail.

4
5
Holmes

Re: Backups

Let this be a lesson to you, boys and girls: off-line, off-site backups were invented for a very good reason.

5
0

Re: Backups

"They had perfectly good backups - the backups didn't fail, someone with the authority to do so deleted them! "

The backups DID fail. An off site backup is not "backup in another machine". An off site backup is "this backup of my data would survive, even if the whole datacenter got burned down in an attack".

This is an off line backup.

They could, for the sake of argument, have used Amazon to run the business - and another cloud provider (just to stay on the cloud) to keep the off line backups. I don't know. Azure? Google? RackSpace? Don't know, don't care - as the point is: it should be in ANOTHER company, not the same. And another location, of course.

2
2
Facepalm

Re: Backups

They had perfectly good backups - the backups didn't fail, someone with the authority to do so deleted them! They could've stored their backups in different locations across different servers but if you let someone get in and delete the backups, that is a process fail, not an infrastructure fail.

the main point is that true off-site backups cannot be accessed via any sort of wire or radio signal... off-site backups are exactly that... off-site... that means manually placed there in their fireproof box and manually removed from there when the next set is put in OR they are needed for disaster recovery...

5
1
Anonymous Coward

Re: Backups

"An off site backup is [something you can do a restore from]"

10/10.

Backups are no good if the restore might not work.

The guys/gals in this picture don't quite seem to have cottoned on to what backups are for.

Backups are so that you CAN do a restore. With some reasonable degree of confidence proportionate to the value of the data at risk, etc.

1
0
Anonymous Coward

actually the S3 team has a pretty good chance of getting everything back but I don't know how often and what they were storing in S3. EBS not so much.

1
0
Anonymous Coward

Code Spaces : Is Down!

http://www.codespaces.com/

0
0

Re: Code Spaces : Is Down!

Downtime notice in Pastebin! These guys are pretty hardcore into the cloud thing.

2
0
Silver badge

Re: Code Spaces : Is Down!

They didn't put it into pastebin - that's just the anon coward playing a slight of hand with the posted url (link doesn't go to what the text implies)

0
0

How lovely, so now disgruntled employee on his/her last day can take down whole company and permanently turn off the lights leaving the building. It's in the cloud folks, and it'll be there until wind blows it away.

13
1

"How lovely, so now disgruntled employee on his/her last day can take down whole company and permanently turn off the lights leaving the building. It's in the cloud folks, and it'll be there until wind blows it away."

And that is not unique to the cloud either. There are many accounts of disgruntled employees scheduling "dd if=/dev/zero of=/dev/sda bs=1m" to run on all in-house servers as they are leaving. Nothing new here. Plus, I've been called into to investigate a hack on in-house servers, where the attackers deliberately wiped Active Directory and IIS metabases on all servers. Because once you get into the domain, you get into all servers on the domain. That company also lost their entire business running on those servers, because while they had backups, reconstructing the configuration took two full days, by which time there were no customers left.

14
0
Bronze badge

And that is not unique to the cloud either.

True, but shouldn't we then be advancing towards making these kinds of criminal mismanagement harder, rather than easier?

3
0
Silver badge

I doubt that there are that many accounts of disgruntled employees destroying their company when they leave. That is a criminal act and you go to jail for it.

And once you have a criminal record, you can say good-bye to any position higher than flipping burgers.

That's a high price to pay for a bit of disgruntling. There may be many people wishing that they could, but I really don't think there are that many who actually do it.

4
3

I doubt that there are that many accounts of disgruntled employees destroying their company when they leave. That is a criminal act and you go to jail for it

Sure, but the key question is how long would you go to jail if you plead guilty at the first hearing, and offered a clean previous record as mitigation? Only 1/3rd of whatever the sentanc was originally.

This becomes important because....

And once you have a criminal record, you can say good-bye to any position higher than flipping burgers.

Certainly you'd struggle to find work if you declared your conviction to an employer. If you didn't, and they failed to do appropriate background checks (smaller companies often fall down here), then you're fine.

Additionally, it appears the maximum jail time under the computer misuse act would be 10 years. Its exceptionally unlikely that you'd be given that as a starting tariff, so lets say 6 years. 1/3rd of that is 2 years. The rehabilitation of offenders act means you won't have to declare it after 4 years.

That's a high price to pay for a bit of disgruntling. There may be many people wishing that they could, but I really don't think there are that many who actually do it.

It is still a high price to pay, but it's not as high as it seems. You could readily fill a 4 year career break with a masters degree and a little backpacking holiday. Obviously, you have the 2 years inside to do as well, but it means you're only out of the game for 6 years total, rather than the rest of your days as you'd (rightly) expect for taking down your employer.

4
1
Anonymous Coward

Re: disgruntled employees

"I doubt that there are that many accounts of disgruntled employees destroying their company when they leave. That is a criminal act and you go to jail for it."

Maybe.

On the other hand, the organisations that got (nearly) taken down frequently won't want their dirty laundry washed in public. When I've seen similar things happen (occasionally over multiple decades), none of them have involved criminal action, they've all been kept as quiet as possible.

4
0

RE: background checks

Certainly you'd struggle to find work if you declared your conviction to an employer. If you didn't, and they failed to do appropriate background checks (smaller companies often fall down here), then you're fine.

Not just small companies! I used to work for a financial services company (it was a subsidiary of a major UK bank) that had a new hire that was given to me to bring up to speed on the tech that we were using on his first morning. He was marched off the premises by security about two hours later only because someone in the office recognised him. He had done time (not for a tech-based misdemeanor) and then changed his name by deed poll and it wasnt picked up in the background checks. Presumably he forgot to mention his name change or time in chokey in the application. They might have increased their security checks now...

1
0
Bronze badge

Re: RE: background checks

There are lots of inappropriate backgrounds checks as well. Many companies do them because everybody else is doing it and it's fashionable. I owned a small company for many years and employed dozens of people. I never ran any background checks and only sacked one employee for being a bad apple. I had suspicions about a couple of employees, but they did good work so what the hell?

I just told a company that was to hire me as an independent contractor to FO when they insisted on signed permission to do a full background check. I wan't going to be handling money in any way other than cashing the checks they paid me with, so why do they need my credit report? They weren't going to supply a company car or auto insurance, so why do they need my driving record? Their decision to use me was not based on having a degree, so why do they need my college transcripts? I wonder if they asked the plumber to sign those forms before doing any work or did they let him get on with unblocking the loo.

Aside from lying on his application, was the offense this bloke got sent up for in any way relevant to his employment with the company? I've never been handed an job application that asked about a name change. In the US, I think that one only has to fess up to felony convictions if asked.

4
2

re disgruntled employee

I recall an episode from the 1970s or thereabouts - insurance company fired their tape librarian, gave her two weeks notice. She spent the next two weeks systematically erasing tapes that contained their entire database of customers, policies. They had no way to know who was a policyholder, what policies they had. They had to go back to their field agents all over the country and ask them to reconstruct the data from their own (paper) files.

1
0

bummer...

Surely a call to Amazon could have resulted in a total lockdown once they knew they were under threat? A freeze of snapshots and an inability to remove history would have been a good place to be, if such things are possible in AWS...

Seems like they tried to fight the attackers single-handed and lost?

Shitty turn of events tho.. there's some right c*nts out there, sure enough..

13
0

Re: bummer...

My thoughts exactly! Also, if a "user" suddenly goes postal on their own data, maybe have something set up to pretend to delete it while alerting the AWS meat sacks to call their client meat sacks on Ye Olde PSTN?

3
0
FAIL

Re: bummer...

Sensible comment, why did you destroy it with the misogynist ending?

0
38
Gold badge

Re: bummer...

That's what you get out of that. "OMG misogynist!"

*sigh*

This is why I hate humans.

32
1
Silver badge

@Jan 0

He might be Australian. We use "c*nt" colloquially, the same way Brits use "bastard" and Americans use "asshole", and it isn't intended to be misogynistic.

21
0
Silver badge

Re: bummer...

Of course they tried to manage through this alone. If they had pulled it off then the damage from the untested update would have been minimal and their internal processes would have been updated to ensure no repeat occurrences of such events.

They really didn't have a choice but to go it alone. They're far to small of a company to brush off the credibility damage of such an event. The ship would have sunk anyway if the scope of the problem even became known. Small specialty firms don't get the luxury of making basic errors like big companies do. There are no failure buffers in small companies so if you fuck up big you've really got an enormous problem. The repercussions go direct to the customer and they simply can't depend on you after that magnitude of failure.

I feel sorry for the whole lot of them, and their customers, I really do. But if you're going to put so very much of your company out of your control then shit like this is going to happen sometimes. There's just no getting around the laws of averages and a certain percentage of all 'middlemen' simply aren't going to be up to the task. The question now is how to prevent this in the future. People will throw money at you if you've got a workable solution to that.

6
1
Gold badge

Re: @Jan 0

Aye, and it isn't limited to a geographic distribution. I use the terms "cuntweasel" and "cockferret" rather often with zero overtones of misandry or misogony. They rank along side "douchepopsicle" and "gonadgremlin" in my lexicon. I don't care if some person or another takes offense beyond the obvious "this is an epithet". If they read deep hatred for $identifiable_group into that then it is entirely because of their own personal hangups.

I'm strictly egalitarian. I hate everyone equally, regardless of gender, race or so forth. Bunch of gonadgremlins, the lot of 'em!

20
2
Bronze badge
Alien

Re: @Jan 0

@Jan 0

>He might be Australian. We use "c*nt" colloquially, the same way Brits use "bastard" and Americans use "asshole", and it isn't intended to be misogynistic.

And context should have told you that, Jan0. "cunt", as used by non-antipodeans, doesn't usually convey--and isn't ever intended to convey--'meany', 'bastard', 'bad person', 'Doctor Evil', etc., so your interpretation of its use as misogynist suggests that you perceive the word to be wholly appropriable, or to have a particular or singular use, and to have a narrow definition. I've found myself reminding a visitor from 'down there', though, that the word is used in these parts differently, when she was expressing anger at having had her bicycle stolen and referring to the thief as a "cunt". Different strokes...

2
1
Bronze badge

Re: bummer...

If the comment said the criminals were "pricks", would you assume that the author hated men?

7
0
Silver badge
Thumb Up

Re: @Jan 0

"He might be Australian. We use "c*nt" colloquially, the same way Brits use "bastard" and Americans use "asshole", and it isn't intended to be misogynistic."

Indeed, but then, so do us Brits, and rest-of-world too!

It's not as if the poster was so annoyed with these people that he/she (Unlike Jan0, I'm not making a sexist assumption!) maliciously and nonsensically compared them to a feminine 'front-bottom'.

Rabid feminists do the legitimate cause more damage than sexist men

12
0

Page:

This topic is closed for new posts.