back to article Code Spaces goes titsup FOREVER after attacker NUKES its Amazon-hosted data

Source code hosting provider Code Spaces has suffered the ultimate cloud nightmare, having been effectively forced out of business by the actions of an attacker who managed to gain access to its Amazon EC2 control panel. The devastating incident began on June 17 when Code Spaces – a company that claimed to offer "Rock Solid, …

COMMENTS

This topic is closed for new posts.
  1. John P

    Complete Bastards

    That is all.

    1. Peter2 Silver badge

      Complete incompetent bastards. No decent backups, no disaster recovery plan, no business continuity plan, nothing nada.

      inexcusable for an IT business.

      1. Pascal Monett Silver badge

        But a brilliant example of why one should NOT trust the cloud with strategic data.

        They had their entire business in the cloud, and the cloud went away - with their business.

        As a cautionary tale, it is strikingly effective.

        1. Dr Who

          That's just utter bo**ocks. This has nothing to do with the cloud and everything to do with truly dreadful system administration. It could just as well have happened in a private data centre as on a cloud service. Cloud services may have their faults, but this is categorically not an example of one of them.

          1. Peter2 Silver badge

            This sort of thing can happen in a data centre, but that's a problem with outsourcing in general. It's usually done because outsourcing is cheaper, and it's usually done cheaper because either staff are outsourced to india and they pay them peanuts or you discover that the reason they can provide it cheaper than you can in house despite using the same suppliers is that your in house solution had redundant discs in RAID and backups, and theirs didn't.

          2. Scroticus Canis
            WTF?

            @ Dr Who -

            "Cloud services may have their faults, but this is categorically not an example of one of them."

            Yeh, tell that to the users mate, I'm sure they will agree with you.

            Vapourware and its data - one puff and it's gone. Wouldn't have happened with decent backups which aren't managed on-line. What a complete fail that is.

          3. Mad Chaz

            On the contrary, it is a good example. An internal data center wouldn't have had an easy to use WEB ACCESSIBLE front end you could use to cause all that damage.

            All the crook needed was a single username/password to get in.

            Plus, cloud pushes you to do everything on it, even when it's a bad idea. (See backups in the above article). This creates a single point of failure.

            Sure, the administrators were idiots NOT to have backups elsewhere, but cloud helped push them in that position.

            1. Cipher

              The article sez that "offsite" backups were lost as well. WTF?

              The offsite BUs were connected to the online servers?

              The entire thing sounds like poor management...

          4. JeffyPoooh
            Pint

            "This has nothing to do with the cloud..."

            It does in the sense that an actual old-school IT server room would likely have some actual backup tapes or portable HDDs locked up in a cupboard, or trucked away to a physical (not virtual) off-site backup. When EVERYTHING is in the cloud, then it is susceptible to hackers in the cloud. When your backups are physical, then the hackers have to be physically present (unless they log into your UPS and cause it to burn the building down, only hours before it would have done so anyway).

            Corrective action and lesson learned is multi-layered, including having non-virtual backups that are also available on the Internet.

            1. Marcelo Rodrigues

              Re: "This has nothing to do with the cloud..."

              "It does in the sense that an actual old-school IT server room would likely have some actual backup tapes or portable HDDs locked up in a cupboard, or trucked away to a physical (not virtual) off-site backup. When EVERYTHING is in the cloud, then it is susceptible to hackers in the cloud."

              But it doesn't. Because EVERY business should have off site backups. It doesn't matter if you use the cloud, a datacenter, a colocation or your basement. Ate least ONE copy of the backups should be kept off site - and not accessible by the systems being backed up (ie: the backup can access the systems, to do a restore or a backup - but the systems can't access the backups).

              Yes, the single sign on system that Amazon uses made it easier. No doubt about it. But the absence of off site backups... that was just neglect, not a cloud problem.

              1. First_Drop

                Re: "This has nothing to do with the cloud..."

                Even if they had substantial and appropriate backups, there still would have been massive disruption to their business and their customers. Arguing that it has nothing to do with the cloud ignores this point.

                I think the other important point is the lack of layered security - a single login portal is surely a major flaw, though I don't know if that is 'cloud typical' or not.

                1. Peter2 Silver badge

                  Re: "This has nothing to do with the cloud..."

                  "Even if they had substantial and appropriate backups, there still would have been massive disruption to their business and their customers."

                  Which is why you have a business continuity plan, which is a (tested) plan as to how you are going to continue the business come what may.

        2. NogginTheNog
          FAIL

          Trust the cloud?

          In effect it looks like they trusted their whole business to a single public-facing login point?!

          Fuck me, is this what 'the cloud' has done to layered security?

      2. Anonymous Coward
        Anonymous Coward

        Re: "inexcusable for an IT business."

        But not untypical of the foolish ones who try to convince us that "the cloud" can replace "the competent" and cost far less, shurely?

        1. BillG
          FAIL

          Re: "inexcusable for an IT business."

          Note that in their explanation here: http://pastebin.com/WvtjMe9T

          They state: Upon realisation that somebody had access to our control panel we started to investigate how access had been gained

          It should, instead, have been: Upon realisation that somebody had access to our control panel we took down the website and disconnected all our systems from the internet..

          Although that might be too much work for people working from home.

      3. John Smith 19 Gold badge
        Unhappy

        @Peter 2

        "Complete incompetent bastards. No decent backups, no disaster recovery plan, no business continuity plan, nothing nada.

        inexcusable for an IT business."

        They figured Amazon would handle everything.

        They were wrong.

        S**t happens. Guaranteed. It's not if you will attacked (as a successful business) it's when.

  2. Dan 55 Silver badge
    Facepalm

    Rather irresponsible

    Code Spaces couldn't find $12.99 down the back of the sofa for a TFA key fob?

    https://aws.amazon.com/iam/details/mfa/

    1. JLV

      Re: Rather irresponsible

      Or download a free tfa app to their mobile.

  3. David Moore
    Mushroom

    Bloody hell.

  4. petur
    FAIL

    Backups

    It's not called backups if it's on the same system (or even same building), that is called a copy.

    1. Anonymous Coward
      Anonymous Coward

      Re: Backups

      Yet every day I hear how the cloud and duplicating means the death of tape.....

      1. Peter2 Silver badge

        Re: Backups

        The only death involved with tape is people who aren't using it. Then again, these sort of people are the sort of inept muppets who don't change their tapes, or just leave the same tape in the drive to be overwritten constantly, so if these people had have been using tape then it probably have been written off along with everything else.

        Cynical, moi?

        Ok, I might be mildly paranoid, but I still I find it comforting to know that I have everything needed to recover from the worst possible disaster imaginable sitting off site and offline.

      2. JDX Gold badge

        Re: Backups

        They had perfectly good backups - the backups didn't fail, someone with the authority to do so deleted them! They could've stored their backups in different locations across different servers but if you let someone get in and delete the backups, that is a process fail, not an infrastructure fail.

        1. Marcelo Rodrigues

          Re: Backups

          "They had perfectly good backups - the backups didn't fail, someone with the authority to do so deleted them! "

          The backups DID fail. An off site backup is not "backup in another machine". An off site backup is "this backup of my data would survive, even if the whole datacenter got burned down in an attack".

          This is an off line backup.

          They could, for the sake of argument, have used Amazon to run the business - and another cloud provider (just to stay on the cloud) to keep the off line backups. I don't know. Azure? Google? RackSpace? Don't know, don't care - as the point is: it should be in ANOTHER company, not the same. And another location, of course.

          1. Anonymous Coward
            Anonymous Coward

            Re: Backups

            "An off site backup is [something you can do a restore from]"

            10/10.

            Backups are no good if the restore might not work.

            The guys/gals in this picture don't quite seem to have cottoned on to what backups are for.

            Backups are so that you CAN do a restore. With some reasonable degree of confidence proportionate to the value of the data at risk, etc.

        2. waldo kitty
          Facepalm

          Re: Backups

          They had perfectly good backups - the backups didn't fail, someone with the authority to do so deleted them! They could've stored their backups in different locations across different servers but if you let someone get in and delete the backups, that is a process fail, not an infrastructure fail.

          the main point is that true off-site backups cannot be accessed via any sort of wire or radio signal... off-site backups are exactly that... off-site... that means manually placed there in their fireproof box and manually removed from there when the next set is put in OR they are needed for disaster recovery...

    2. Joe User
      Holmes

      Re: Backups

      Let this be a lesson to you, boys and girls: off-line, off-site backups were invented for a very good reason.

  5. Anonymous Coward
    Anonymous Coward

    actually the S3 team has a pretty good chance of getting everything back but I don't know how often and what they were storing in S3. EBS not so much.

  6. Anonymous Coward
    Anonymous Coward

    Code Spaces : Is Down!

    http://www.codespaces.com/

    1. Ticl

      Re: Code Spaces : Is Down!

      Downtime notice in Pastebin! These guys are pretty hardcore into the cloud thing.

      1. Jamie Jones Silver badge

        Re: Code Spaces : Is Down!

        They didn't put it into pastebin - that's just the anon coward playing a slight of hand with the posted url (link doesn't go to what the text implies)

  7. DainB Bronze badge

    How lovely, so now disgruntled employee on his/her last day can take down whole company and permanently turn off the lights leaving the building. It's in the cloud folks, and it'll be there until wind blows it away.

    1. Tom Samplonius

      "How lovely, so now disgruntled employee on his/her last day can take down whole company and permanently turn off the lights leaving the building. It's in the cloud folks, and it'll be there until wind blows it away."

      And that is not unique to the cloud either. There are many accounts of disgruntled employees scheduling "dd if=/dev/zero of=/dev/sda bs=1m" to run on all in-house servers as they are leaving. Nothing new here. Plus, I've been called into to investigate a hack on in-house servers, where the attackers deliberately wiped Active Directory and IIS metabases on all servers. Because once you get into the domain, you get into all servers on the domain. That company also lost their entire business running on those servers, because while they had backups, reconstructing the configuration took two full days, by which time there were no customers left.

      1. xperroni

        And that is not unique to the cloud either.

        True, but shouldn't we then be advancing towards making these kinds of criminal mismanagement harder, rather than easier?

      2. Pascal Monett Silver badge

        I doubt that there are that many accounts of disgruntled employees destroying their company when they leave. That is a criminal act and you go to jail for it.

        And once you have a criminal record, you can say good-bye to any position higher than flipping burgers.

        That's a high price to pay for a bit of disgruntling. There may be many people wishing that they could, but I really don't think there are that many who actually do it.

        1. TopOnePercent

          I doubt that there are that many accounts of disgruntled employees destroying their company when they leave. That is a criminal act and you go to jail for it

          Sure, but the key question is how long would you go to jail if you plead guilty at the first hearing, and offered a clean previous record as mitigation? Only 1/3rd of whatever the sentanc was originally.

          This becomes important because....

          And once you have a criminal record, you can say good-bye to any position higher than flipping burgers.

          Certainly you'd struggle to find work if you declared your conviction to an employer. If you didn't, and they failed to do appropriate background checks (smaller companies often fall down here), then you're fine.

          Additionally, it appears the maximum jail time under the computer misuse act would be 10 years. Its exceptionally unlikely that you'd be given that as a starting tariff, so lets say 6 years. 1/3rd of that is 2 years. The rehabilitation of offenders act means you won't have to declare it after 4 years.

          That's a high price to pay for a bit of disgruntling. There may be many people wishing that they could, but I really don't think there are that many who actually do it.

          It is still a high price to pay, but it's not as high as it seems. You could readily fill a 4 year career break with a masters degree and a little backpacking holiday. Obviously, you have the 2 years inside to do as well, but it means you're only out of the game for 6 years total, rather than the rest of your days as you'd (rightly) expect for taking down your employer.

          1. Alfie

            RE: background checks

            Certainly you'd struggle to find work if you declared your conviction to an employer. If you didn't, and they failed to do appropriate background checks (smaller companies often fall down here), then you're fine.

            Not just small companies! I used to work for a financial services company (it was a subsidiary of a major UK bank) that had a new hire that was given to me to bring up to speed on the tech that we were using on his first morning. He was marched off the premises by security about two hours later only because someone in the office recognised him. He had done time (not for a tech-based misdemeanor) and then changed his name by deed poll and it wasnt picked up in the background checks. Presumably he forgot to mention his name change or time in chokey in the application. They might have increased their security checks now...

            1. MachDiamond Silver badge

              Re: RE: background checks

              There are lots of inappropriate backgrounds checks as well. Many companies do them because everybody else is doing it and it's fashionable. I owned a small company for many years and employed dozens of people. I never ran any background checks and only sacked one employee for being a bad apple. I had suspicions about a couple of employees, but they did good work so what the hell?

              I just told a company that was to hire me as an independent contractor to FO when they insisted on signed permission to do a full background check. I wan't going to be handling money in any way other than cashing the checks they paid me with, so why do they need my credit report? They weren't going to supply a company car or auto insurance, so why do they need my driving record? Their decision to use me was not based on having a degree, so why do they need my college transcripts? I wonder if they asked the plumber to sign those forms before doing any work or did they let him get on with unblocking the loo.

              Aside from lying on his application, was the offense this bloke got sent up for in any way relevant to his employment with the company? I've never been handed an job application that asked about a name change. In the US, I think that one only has to fess up to felony convictions if asked.

        2. Anonymous Coward
          Anonymous Coward

          Re: disgruntled employees

          "I doubt that there are that many accounts of disgruntled employees destroying their company when they leave. That is a criminal act and you go to jail for it."

          Maybe.

          On the other hand, the organisations that got (nearly) taken down frequently won't want their dirty laundry washed in public. When I've seen similar things happen (occasionally over multiple decades), none of them have involved criminal action, they've all been kept as quiet as possible.

      3. Gary Bickford

        re disgruntled employee

        I recall an episode from the 1970s or thereabouts - insurance company fired their tape librarian, gave her two weeks notice. She spent the next two weeks systematically erasing tapes that contained their entire database of customers, policies. They had no way to know who was a policyholder, what policies they had. They had to go back to their field agents all over the country and ask them to reconstruct the data from their own (paper) files.

  8. lansalot

    bummer...

    Surely a call to Amazon could have resulted in a total lockdown once they knew they were under threat? A freeze of snapshots and an inability to remove history would have been a good place to be, if such things are possible in AWS...

    Seems like they tried to fight the attackers single-handed and lost?

    Shitty turn of events tho.. there's some right c*nts out there, sure enough..

    1. Suburban Inmate

      Re: bummer...

      My thoughts exactly! Also, if a "user" suddenly goes postal on their own data, maybe have something set up to pretend to delete it while alerting the AWS meat sacks to call their client meat sacks on Ye Olde PSTN?

    2. Jan 0 Silver badge
      FAIL

      Re: bummer...

      Sensible comment, why did you destroy it with the misogynist ending?

      1. Trevor_Pott Gold badge

        Re: bummer...

        That's what you get out of that. "OMG misogynist!"

        *sigh*

        This is why I hate humans.

      2. Steven Roper

        @Jan 0

        He might be Australian. We use "c*nt" colloquially, the same way Brits use "bastard" and Americans use "asshole", and it isn't intended to be misogynistic.

        1. Trevor_Pott Gold badge

          Re: @Jan 0

          Aye, and it isn't limited to a geographic distribution. I use the terms "cuntweasel" and "cockferret" rather often with zero overtones of misandry or misogony. They rank along side "douchepopsicle" and "gonadgremlin" in my lexicon. I don't care if some person or another takes offense beyond the obvious "this is an epithet". If they read deep hatred for $identifiable_group into that then it is entirely because of their own personal hangups.

          I'm strictly egalitarian. I hate everyone equally, regardless of gender, race or so forth. Bunch of gonadgremlins, the lot of 'em!

          1. MachDiamond Silver badge

            Re: @Jan 0

            "I'm strictly egalitarian. I hate everyone equally, regardless of gender, race or so forth. Bunch of gonadgremlins, the lot of 'em!"

            …. and their newts.

            1. Trevor_Pott Gold badge

              Re: @Jan 0

              ...but I like newts! I keep all sorts of lizards, and newts are cool!

        2. Michael Thibault
          Alien

          Re: @Jan 0

          @Jan 0

          >He might be Australian. We use "c*nt" colloquially, the same way Brits use "bastard" and Americans use "asshole", and it isn't intended to be misogynistic.

          And context should have told you that, Jan0. "cunt", as used by non-antipodeans, doesn't usually convey--and isn't ever intended to convey--'meany', 'bastard', 'bad person', 'Doctor Evil', etc., so your interpretation of its use as misogynist suggests that you perceive the word to be wholly appropriable, or to have a particular or singular use, and to have a narrow definition. I've found myself reminding a visitor from 'down there', though, that the word is used in these parts differently, when she was expressing anger at having had her bicycle stolen and referring to the thief as a "cunt". Different strokes...

        3. Jamie Jones Silver badge
          Thumb Up

          Re: @Jan 0

          "He might be Australian. We use "c*nt" colloquially, the same way Brits use "bastard" and Americans use "asshole", and it isn't intended to be misogynistic."

          Indeed, but then, so do us Brits, and rest-of-world too!

          It's not as if the poster was so annoyed with these people that he/she (Unlike Jan0, I'm not making a sexist assumption!) maliciously and nonsensically compared them to a feminine 'front-bottom'.

          Rabid feminists do the legitimate cause more damage than sexist men

        4. phuzz Silver badge
          Devil

          Re: @Jan 0

          In Scotland it's practically a term of endearment, and used by both men and women. Think of it as reclaiming the word from the misogynists, and bringing it back to being the generic insult it always used to be.

        5. MachDiamond Silver badge

          Re: @Jan 0

          Just remember you can't say "c*nt" in Canada.

          At least that's what they told Rodney Rude. Saying something like that to a comedian is like handing a loaded shotgun to a drunk redneck.

      3. Anonymous Coward
        Anonymous Coward

        Re: bummer...

        If the comment said the criminals were "pricks", would you assume that the author hated men?

    3. Don Jefe

      Re: bummer...

      Of course they tried to manage through this alone. If they had pulled it off then the damage from the untested update would have been minimal and their internal processes would have been updated to ensure no repeat occurrences of such events.

      They really didn't have a choice but to go it alone. They're far to small of a company to brush off the credibility damage of such an event. The ship would have sunk anyway if the scope of the problem even became known. Small specialty firms don't get the luxury of making basic errors like big companies do. There are no failure buffers in small companies so if you fuck up big you've really got an enormous problem. The repercussions go direct to the customer and they simply can't depend on you after that magnitude of failure.

      I feel sorry for the whole lot of them, and their customers, I really do. But if you're going to put so very much of your company out of your control then shit like this is going to happen sometimes. There's just no getting around the laws of averages and a certain percentage of all 'middlemen' simply aren't going to be up to the task. The question now is how to prevent this in the future. People will throw money at you if you've got a workable solution to that.

      1. Paul Smith

        Re: bummer...

        Bullshit! Big or small doesn't matter. They had no disaster recovery plan, so they had no chance of recovering from a diaster.

        Remember that the next time you think of trusting someone else with anything you own or are responsible for, if it is important enought to worry about losing, then you must have it is more then one place, and that those places must not be reacheble from each other.

  9. asdf

    wait a minute

    Who in there right mind would ever store a company code repository in the cloud? A very large portion of the value of many software companies is the bits sitting is said repository. Oh yeah some jag off bean counter type VP who has suddenly figured out how to save the company a few dollars. Stupidity rewards itself unless it has a golden parachute.

    1. MachDiamond Silver badge

      Re: wait a minute

      It's the new way of doing business.

      Everybody is doing it.

      If you aren't in the cloud, you falling behind.

      Why not outsource your data and computing to the cloud and save some money?

      Tape is dead.

      All of your data is being backed up in multiple data centers around the world. You're covered.

      It's more secure to keep your data in the cloud.

      Stop me when you haven't heard one.

      Now that you've discovered your company assets are being held for ransom and might be trashed, it's a great time to enjoy a few hours of music and new product announcements while you are on hold with (insert your cloud provider here). There is always the chance that the person that finally answers the phone will speak your language without a horrendously thick accent.

  10. xperroni
    Coat

    And the "understatement of the year" prize goes to...

    "On behalf of everyone at Code Spaces, please accept our sincere apologies for the inconvenience this has caused to you (...)."

    I can only imagine these people getting to work one day, finding the whole company building burned down to the ground, and then sighing with typical British detachment, "well, that's going to be inconvenient".

    1. Stevie

      Re: And the "understatement of the year" prize goes to...

      "I can only imagine these people getting to work one day, finding the whole company building burned down to the ground, and then sighing with typical British detachment, "well, that's going to be inconvenient"

      Well, they took my red thtapler, and when the cake wath being handed out, I didn't receive a piethe.

  11. Adam White

    Nothing says credibility...

    ...like exclaimation marks in a press release.

  12. Mark 85

    I would like to think...

    that any company using them for a repository isn't using it for live code or code in development but as a backup. Oh wait.. bean-counters tell everyone where to put stuff... nevermind.

  13. Feldagast

    Why I don't trust the cloud, where your dependent on others or a internet connection to work or see your data. How is it anything other than a revenue stream for these companies? If I don't back up my files on my computer its my fault, letting them sit out there is another thing.

  14. Franklin

    I used to think that any data you store in "the cloud" exists on someone else's whim if you store it there for free.

    Apparently it exists on someone else's whim if you pay to store it there, too.

    1. xperroni
      Facepalm

      Welcome to the cloud

      Where your entire operation can disappear overnight and being a paying customer guarantees nothing whatsoever.

      1. Anonymous Coward
        Anonymous Coward

        Re: Welcome to the cloud

        "Where your entire operation can disappear overnight and being a paying customer guarantees nothing whatsoever."

        Yep, It certainly can if you elect to store it on one platform, from one vendor, with one unified interface to access all the services, including the ones used for 'backup'. I don't have a problem with the cloud per se, I do have a problem with the bizarre faith in it that seems so prevalent. If you're local, take it to tape, then tapes offsite. If you're cloud-based, take it to another cloud provider.

        You have data, you back it up elsewhere - if it really matters to you. Simple in theory.

        1. xperroni
          Facepalm

          Re: Welcome to the cloud

          I don't have a problem with the cloud per se, I do have a problem with the bizarre faith in it that seems so prevalent.

          But you see, "safe if properly handled" is just as true for handguns, yet we don't allow everyone to have one, correctly inferring most people won't.

          Cloud hosting services are the same – a reasonable proposition for a world where people can be relied upon to not be goddmaned stupid, which unfortunately isn't ours.

          1. Pascal Monett Silver badge

            Re: just as true for handguns

            Yeah, but guns come with license backed by government law and if you don't respect that you go to jail, whereas cloud comes with marketing spiel and beancounter approval and if you don't listen to them you lose your job.

            Now we know that if you do listen to them you can lose your company, or at least your data.

            The conclusion is : never listen to a non-technical person on technical issues if you can help it.

            1. TopOnePercent

              Re: just as true for handguns

              Never listen to a non-technical person on technical issues

              This should be stencilled in foot high letters on the wall of every manager in every company. It could be accompanied by other epithets such as:

              Your teenager is not an appropriate technical person.

              If the people reporting to you don't think you're a technical person, then you're not a technical person.

          2. Anonymous Coward
            Anonymous Coward

            Re: just as true for handguns

            U.S. company - the cloud is safe, and handguns are mandatory.

  15. lambda_beta
    Linux

    Anyone who stores anything in the "cloud" deserves what they get. I'm not sure how this out-sourcing thing got started, but to put your source code on any server but your own is complete stupidity.

    1. TopOnePercent
      Alert

      Anyone who stores anything in the "cloud" deserves what they get.

      I store a bunch of personal stuff in cloud-like services such as dropbox, google drive, sky one drive etc. Its all encrypted with TrueCrypt, and copies reside on the hard disk of each of my computers.

      Obviously it wouldn't be appropriate for a business to store its data in this manner, but for individuals, the cloud can provide free offsite backups of things. Lets face it, it beats mailing a DVD of data to my parents house!

  16. Anonymous Coward
    Anonymous Coward

    I now see the big advantage of the cloud...

    You don't even have to hire a skip to throw everything into when you bork the company.

  17. willi0000000

    i did some non-IT work for a large insurance company some 40 years ago and, because the work involved scheduling things like power outages, i learned how their backups worked. they had on-site backups right in the same building, another copy in a small building across a very large parking lot and a third set warehoused, sort of, locally.

    they also had three additional days of backups that were constantly in-transit. day zero left the company (in the northeast US) for the west coast as soon as they were completed, day minus-one began moving from the west coast to the Gulf coast at the same time and day minus-two was simultaneously shipped home from the Gulf coast for overwrite.

    the philosophy was that a small fire could take out the building copy or a very local event (tornado) could take out the two backups at headquarters but the warehoused copy might survive. if that copy was lost due to a larger scale event (flood, hurricane, earthquake) they still had three copies circulating around the country (moving targets, so to speak) that would take nothing short of nuclear war or the fall of the government to lose.

    they also paid a hefty price for a "pending" order for equipment* to replace hardware at, literally, a moment's notice and a leased space with power, HVAC, etc.

    *warehoused or next items off the assembly line, no excuses.

    [and you think you're paranoid . . . these folks had belt, suspenders and the pants glued and stapled to their waist with several spare pair handy, and spare legs!]

    1. Terry 6 Silver badge

      [and you think you're paranoid . . . these folks had belt, ....

      And so it should damn well be.

      Any organisation that relies on data needs to have copies of that data kept physically separated from the machines that created ( read, able to make changes to ) it. Ideally in a different building. Stuff happens.

      When I worked for a local authority special needs service I made sure there were automatic and frequent manual backups of our main database of pupils, into a zip file, but that was also backed up to DVD. Some of these were then also stored elsewhere around the network, like on a local HDD of a user machine . Termly at least there was a copy encrypted and taken offsite to my and/or the admin officer's homes.

      The database developer kept his own copies of the actual engine, with backups.

      Staff reports and other child data were kept on the server, I spent many years trying to get our lords and masters to develop a way to get a copy stored at the council offices, but sadly that never got anywhere. And after many years of arguing I at least got the Local Authority team to create a system of backups to a pair of portable HDD, that were swapped weekly and one of which was always kept in a safe. In another room at least. Not as good as them carrying it offsite, but the best I could get. Many staff kept our own copies of our work on encrypted memory sticks, just in case.

      This might sound paranoid, even in the light of this article. But imagine the consequences if our server was wrecked in any way.

      It might be a small amount of data compared to most places, but these were about supporting real kids with special needs.

      Hundreds of kids' notes with issues ranging from tracking learning difficulties to concerns about abuse could have been lost. Kids would have suffered, and at the very least our credibility would have been hit. (Not too badly, since blaming the computer still works in the public mind.)

      1. Ben Norris

        Re: [and you think you're paranoid . . . these folks had belt, ....

        Ah but copying around USB sticks is how everyone keeps getting their personal data stolen. Backups need to be offsite but they also need to be kept secure both there and in transit.

        1. This post has been deleted by its author

        2. Terry 6 Silver badge

          Re: [and you think you're paranoid . . . these folks had belt, ....

          And towards the end of my work there I was finding it really hard to find sticks with built-in encryption (let alone at a decent cost ). Suddenly they became hard to get and really expensive.

  18. Stuart Castle Silver badge

    Something to be said for tape..

    This is the problem with a lot of these free or even cheap services. They save money by not having adequate backups in place. After all, backups cost a lot of money and if everything is working properly, you'll never use them, so they don't contribute anything to the bottom line right?

    We have network accessible storage areas at work, with limited space for each user (and quite a low limit). I am a sys admin, but don't have responsibility for that side of the network, so when a user asked me why their space was so limited, I checked with the sysadmins who are, pointing out that storage is relatively cheap. Apparently the problem was not purchasing the storage, but the cost of backing up (we use a full multi-generational backup system with on and off line backups off site, at least two of which are not even in London) and storing the tapes.

    It does apparently cost us a lot of money to do the amount of backing up we do (even though we really back up the bare essentials of what we need to, so could spend a *lot* more), but we've balanced that against the amount we'd lose if we didn't backup. It's unlikely we'd be able to continue trading if there were a major disaster or failure and we had no backups.

  19. Daggerchild Silver badge
    Alert

    But... but... *Cloud*!!

  20. a_yank_lurker

    How did they get access

    I am not very familiar with AWS security protocols. I would assume at a minimum they require a password if not more. So for someone to get access to the control panel implies they either got the password by spear-phishing or it was inside job.

    1. Flywheel

      Re: How did they get access

      There's a username and password, plus some form of 2FA which can be done on a hardware dongle (which seem to die when the battery runs out) or via Google Authenticator. Sounds like someone had a leetle too much credentials knowledge..

    2. Anonymous Coward
      Anonymous Coward

      Re: How did they get access

      More likely they had their login details saved on their local pc and picked up (or were sent) a trojan.

  21. Neil B

    This is a process failure, not a "cloud" failure. It seems these guys established their business by basically lying about the security they offered, and now it has collapsed around their ears.

    1. Androgynous Cupboard Silver badge

      Well, given the failure in process was relying on the cloud... splitting hairs a bit aren't you?

  22. Medixstiff

    This is why...

    We need tougher prison sentences, it's not just the company that suffers, it's the customers too, they tried doing the right thing by going to a third party service, they cannot tell exactly how that service behaves in the event of an attack like this, they expect the company to know what they are doing.

    I remember seeing photos of what happens to people who get the cane in Indonesia and wasn't exactly surprised at the low numbers of repeat offenders. I expect a nice little hacker safe and sound thinking he's the bees knees for destroying someone so utterly wouldn't repeat offend if his backside was missing a few layers of skin.

    That would be on top of a fine to be paid over time, similar to a HECS debt, that way they cannot file for bankruptcy, the fine would follow them until paid off or they die.

  23. TopOnePercent

    Good luck to those affected

    Did the company get it right? No. And they really should have.

    That doesn't mean everyone in the company deserved to lose their jobs - I'd imagine the HR staffer, receptionist etc lacked the technical knowledge to know they'd got it wrong. It may not even be the fault of their BOFH - be honest guys n girls, do your managers implement all of your recommendations? No, mine doesn't either.

    So best of luck to the staff who are to lose their jobs, and also to the customers affected - hopefully they have additional backups or onsite copies.

  24. Benjol

    Another advantage of git...

    Is that you have an 'automatic backup' of all your history locally.

  25. ForthIsNotDead

    What about Amazon?

    How did the attacker gain access? If the credentials weren't leaked from inside the company, then Amazon has some questions to answer. This has to reflect badly on Amazon, too.

    1. Neil B

      Re: What about Amazon?

      @ForthisNotDead Law of averages says social engineering hack. You don't think Code Spaces would be trying to point the finger at Amazon if they had even the tiniest evidence it was their fault?

  26. Anonymous Coward
    Anonymous Coward

    Never mind cloud

    It's bad enough that a business should entrust its entire crown jewels to a cloud storage service. But in this case, unless I've misunderstood, it seems like a whole bunch of companies entrusted their crown jewels to a third party who then stored said jewels on their behalf ... in something with a single point of failure.

    That's a whole extra level of fuckwittery IMHO.

  27. Speltier

    Take Home

    The take home is that AWS/Azure/thunderhead/etc should offer the option for mandatory delays to backup deletes. If the backups have built in delays for deletes (delays in DAYS or WEEKS, not MICROSECONDS), it gives meat bags enough time to recover when the barbarians have breached all the outer and inner defenses and are running terminally rampant in the CEO's concubinorium....

    (one should also back the data to a different cloud outside the EMP radius of your chosen cloud, if one is fixated on using remote services. Indeed choose a cloud in some remote area of Africa which might be subject to genocide, but so poor no one would waste more than a few tonnes of chemical destruction vs. an expensive nuclear EMP generator. That way when the the first world is gone, your data will survive.).

  28. Stretch

    someone do this to github please

  29. DropBear
    Trollface

    Well, this has to take the cake for most fitting the line "all your base are belong to us"...

  30. disk iops

    S3 is lazy delete. Just because you delete stuff in the console does NOT mean it's actually gone. You have ~3+ days to get it back. EBS snapshots are stored in S3 AFAIK. The first and gravest mistake (aside from using cloud services in a sloppy manner in the first place) was to not IMMEDIATELY call AWS support and get the account administratively locked. They handle loss of account control routinely.

  31. Anonymous Coward
    Anonymous Coward

    Buyer beware

    Don't forget the collapse of 2e2 as well. Then the customers were "blackmailed" in to paying up to £40k each to keep the service running for a couple more weeks and give them time to migrate.

  32. KHobbits

    Wise men enable security

    No matter what sort of solution you decide to go with, you are throwing your trust behind a vendor, that could be the guys that make your tape drive, or the guys who are providing you with offsite hosting.

    Amazon has a fairly decent track record, so I'm fairly happy to trust them with my data.

    AWS has a rather decent level of security. If you want to have a web application interact with the AWS API you set it up with just enough IAM permissions to do the task it needs to do. You can bake these credentials into the app, or into a server, or even a set of credentials for each app per server.

    You should never need to write your console credentials in any script or server location. Each console account can be fine tuned in privileges. I for example have granted my development team with access to view all my servers, ssh to them with the key attached to their account, but they can only use sudo and deploy to the development stacks.

    The developers have read only access to the s3 object storage, but not access to delete, and don't have any access to the AMI's or ability to manage databases what so ever.

    Me and my boss both have our own admin console credentials, protected via 2 factor auth, with the root credentials secured in the same manner.

    This is all standard, and recommended behaviour for any AWS account. Anyone that isn't doing this is ignoring recommended practices, and if you would be stupid not to if you are hosting anything worth protecting.

  33. BongoJoe

    "In summary, most of our data, backups, machine configurations and offsite backups were either partially or completely deleted."

    Surely off site backups should be off line and stored somewhere very, very secure. Or am I doing things needlessly the old-fashioned way?

  34. Anonymous Coward
    Anonymous Coward

    Learn from it :-I

    The problem was not the cloud, the cloud was protected, however the keys to the kingdom were not protected.

    We should all learn from this:

    * Review what can be done when suspicious activity takes place, ie production servers being deleted at will.

    * How do we protect backup data on disk. Pehaps two phase authentication

    I feel bad for all the people affected by this attack.

  35. okcomputer44

    Why only 1 account?

    I guess they made a mistake to operate the whole lot within just one account.

    The whole design should be made like 1 sysadmin could operate only portion of the cloud not the whole lot.

    Imagine if someone had a bad day or just want to make some fun?

    Like the guy in Braking Bad whose daughter died because of the drogs and he makes a mistake at the air plane centre and 2 air plane collides and loads of people die.

    Also would I trust in anybody and give access for everything?

    Hell no way! Clearly they would not think of that this could happen.

    Why would you need to operate the whole lot from just one account anyway?

  36. si063

    Lessons:

    1) Do not trust your entire business to a single publicly accessible login point (in this case amazon's web console).

    2) Make sure you have secure backups, that's backups which cant just be deleted with a click.

    3) Consider modes of failure, when X Y or Z service or product or security mechanism fails, what is your position?

    In summary... this is what happens when developers are left to do infrastructure.

This topic is closed for new posts.

Other stories you might like