back to article Bloke clicks GitHub 'commit' button in Visual Studio, gets slapped with $6,500 AWS bill

A web developer from South Africa said a bug in a tool for using Microsoft's Visual Studio IDE with code-sharing site GitHub inadvertently exposed his sensitive data – and the error cost him more than $6,500 (£4,250) in just a few hours. Carlo van Wyk of Cape Town–based Humankode said he used the GitHub Extension for Visual …

  1. Anonymous Coward
    Anonymous Coward

    almost makes you miss VS6

    But but but the cloud and SaS is where its at. Especially with often a software companies crown jewels the source code. Bonus is also getting developers to have to sign into their accounts on the internet to do anything (data sharing (selling) is caring).

  2. YetAnotherLocksmith Silver badge

    How about...

    ...writing an extention that scans user code for AWS keys *before* it uploads, & alerts you if it would be publicly exposed?

    1. Goldmember

      Re: How about...

      In this case it may not have flagged it up, as the repository it was supposed to go into was private.

      But yes, if the plugin scans for AWS keys (full stop) then simply says 'don't do it' upon finding any, it would be a good idea.

      On another note, the article doesn't mention who now foots the bill for this. Will GitHub be paying AWS for this cockup?

      1. Anonymous Coward
        Anonymous Coward

        Re: How about...

        On another note, the article doesn't mention who now foots the bill for this. Will GitHub be paying AWS for this cockup?

        In the past Amazon has waived the charges, I'd expect them to do the same here (especially since the charge is so trivial).

        1. Someone Else Silver badge
          Mushroom

          Re: How about...

          In the past Amazon has waived the charges, I'd expect them to do the same here (especially since the charge is so trivial).

          Must be nice up in that ivory tower you 1%'ers live in, but down here on earth, $6500 is real money and is hardly considered a "trivial" charge.

          1. jonathanb Silver badge

            Re: How about...

            Amazon lives in your aforementioned Ivory Towers, and it is trivial for them. It is also trivial compared to what most of their customers spend, who tend to be companies rather than individuals.

    2. Anonymous Coward
      Anonymous Coward

      Re: How about...

      Usernames and Passwords is another thing which people frequently feel that it is ok to put in source code -- this is the only way they will learn

    3. Mark 65

      Re: How about...

      ...using .gitignore before you commit anything?

  3. Michael Hoffmann Silver badge
    Happy

    Real men use command line

    Here I semi-guiltily felt like a bit of a curmudgeon and dinosaur for always switching back to a Cygwin command line session to do all Git SCM from there instead of the VS built-ins.

    That icon is intended to be a "smug face".

    1. John Sanders
      Trollface

      Re: Real men use command line

      >> That icon is intended to be a "smug face".

      You mean this one?

      It is the one I got when I RFA, and I thought the same as you.

      1. Michael Hoffmann Silver badge

        Re: Real men use command line

        I thought that was troll face, not smug face. You kids and your memes...

    2. Michael Wojcik Silver badge

      Re: Real men use command line

      I do everything I can from the command line. I only fire up Venomous Studio when I have to debug managed code (WinDbg still isn't great at that, and mdbg is OK for some purposes but not great with complex systems that have lots of interacting processes), or investigate a customer issue that's specific to the IDE.

      But then I've never liked GUI clients for change-management systems. Or for most other things.

  4. x 7

    this sounds like deja vu.........wasn't there a similar incident around 12 months ago?

    1. Haku
      1. Destroy All Monsters Silver badge
        Coat

        The voice form the bathtub

        "This happened before ... and it will happen again"

  5. Ole Juul

    as we push the big, red "Publish" button on this story

    Can we have a picture of that?

    1. Anonymous Coward
      Pint

      Re: as we push the big, red "Publish" button on this story

      Even better would be to have one made with bluetooth so everyone could have an Official El Reg Publish button.

    2. Herby

      Re: as we push the big, red "Publish" button on this story

      Picture? How about this...

      http://www.thinkgeek.com/product/15a5/?pfm=Search&t=big%20red%20button

      Sounds like it fits the bill. Sorry it isn't official Reg material.

    3. Dan 55 Silver badge
      Mushroom

      Re: as we push the big, red "Publish" button on this story

      How about a video?

      https://www.youtube.com/watch?v=NITBfc1EOBo

      What happens in Vulture Central before an AO article is published.

    4. Vic

      Re: as we push the big, red "Publish" button on this story

      Can we have a picture of that?

      Here you go.

      Vic.

      1. Steven Raith

        Re: as we push the big, red "Publish" button on this story

        Surely this is more apt?

        The Emergency Party Button

  6. Mark 85

    The missing moral...

    Don't trust the cloud or SaS. Period.

    1. Anonymous Coward
      Anonymous Coward

      Re: The missing moral...

      You are looking at the cloud.

      You publish your stuff to world & dog, however inadvertently, on the cloud.

      The cloud is looking at you.

      ...but it's not the cloud's fault.

      1. Androgynous Cupboard Silver badge

        Re: The missing moral...

        Man, that is deep.

      2. Anonymous Coward
        Anonymous Coward

        Re: The missing moral...

        >You publish your stuff to world & dog

        Unless you care about things like privacy or I don't know not exposing your employer's code that you don't own.

  7. Anonymous Coward
    Anonymous Coward

    Free account monitoring service for the next year...

    I guess since GitHub (and MS?) are responsible for the disclosure of his information, they'll be offering him free credit account monitoring for a year...

    Oh what, they're not even doing that?

    1. Anonymous Coward
      Anonymous Coward

      Re: Free account monitoring service for the next year...

      "I guess since GitHub (and MS?) are responsible for the disclosure of his information"

      He is also responsible. Anybody who posts anything with commercially sensitive data to a remote location without encryption is to say the least somewhat careless. I wouldn't wish that kind of attack on anybody, but, wtf was he doing? You always have to consider the possibility of fat finger leakage.

      Having said that, if the guy at my former company (who didn't want separate configuration files because it was "too complicated" and so wanted embedded credentials baked in) is reading this - now do you believe me?

      1. JDX Gold badge

        Re: Free account monitoring service for the next year...

        But if they got _hacked_ and his key stolen, wouldn't they be liable?

      2. Anonymous Coward
        Anonymous Coward

        Re: Free account monitoring service for the next year...

        "I wouldn't wish that kind of attack on anybody, but, wtf was he doing?"

        Probably just another idiot millenial who thinks The Cloud is a magic place in the sky running on fairydust and unicorns tears where nothing bad ever happens, rather than just someone elses computer with all the attendant risks.

        Any private code our company has goes nowhere near a public computer of any form. Its on our private servers and is backed up to tape every 48 hours. End.

        1. Anonymous Coward
          Anonymous Coward

          Re: Free account monitoring service for the next year...

          >Probably just another idiot millenial

          Can't resist. What do you mean dude they they didn't teach your computer classes in Ruby on Rails? They even made us go bare metal with C# and Java for one class.

          1. Someone Else Silver badge
            WTF?

            @AC -- Re: Free account monitoring service for the next year...

            They even made us go bare metal with C# and Java for one class

            Thanks AC...I needed a good guffaw for the middle of the week.

      3. Anonymous Coward
        Anonymous Coward

        Re: Free account monitoring service for the next year...

        > posts anything with commercially sensitive data to a remote location without encryption ...

        Don't put any keys or passwords in source code -- even encrypted -- the just shouldn't be a need ever.

        1. JDX Gold badge

          Re: Free account monitoring service for the next year...

          Really? Where do you want to put them - in a config file or a DB? But then the config file is in plain text and you can access the DB unless it requires authentication. Oh, but where are you going to put the password to access the DB?

          1. Gerhard Mack

            Re: Free account monitoring service for the next year...

            Config files should NOT be synched with GIT. They should be local to the server in question and if it contains passwords should have it's rights restricted. That is basic security practice.

            1. JDX Gold badge

              Re: Free account monitoring service for the next year...

              So you don't back up your config files? If they're not in Git they're going to be somewhere...

              1. Gerhard Mack

                Re: Free account monitoring service for the next year...

                Not in Git, git is a source code management system and not a backup. If anything, I have an example config with a different name (config.distrib) otherwise you pollute git with a ton of changes to config files, and it gets worse when you have multiple conflicting changes (dev vs live/server 1 vs sever 2 etc).

                Passwords, keys or any other private info should not be stored in Git, instead, they should be in a proper backup system.

  8. Phil Endecott

    Trawling

    Has this word now been lost?

    Do we now call fishing boats "trollers"?

    1. maffski

      Re: Trawling

      Do we now call fishing boats "trollers"?

      It might depend on whether said fishing boat is trawling or trolling.

      1. Destroy All Monsters Silver badge

        Re: Trawling

        That would be thrauhlerz in phreakspeak... (apparently a derivative of Lovcraphtianesque)

      2. TeeCee Gold badge
        Coat

        Re: Trawling

        Depends whether or not the crew have internet access and nude pics of you....

      3. BitDr

        Re: Trawling

        Or better yet, it depends on whether it is a fishing boat or a phishing boat. Are phishing boats crewed by trolls?

  9. Anon Adderlan

    This is the kind of thing that keeps me up at night

    If 'I'm' buying over a thousand dollars worth of instances, then I want Amazon to confirm it was actually ME before I'm billed. I also want to set limits on what and how much an AWS key can be used for. I can already do both with credit card numbers, so what's the deal here? This kind of theft seems ridiculously easy otherwise.

    1. This post has been deleted by its author

    2. Daniel Voyce

      Re: This is the kind of thing that keeps me up at night

      You can set limits on what an AWS IAM key does, it sounds like he has put in a root IAM key into his code which basically gives full control over things like this when all he possibly wanted was to be able to access an S3 bucket.

      There are also perfectly valid use cases where one might want to spin up a huge memory instance and then shut it down after a job completes, that is one of the benefits of AWS is that it can respond to changing conditions - having a confirmation required in each of these cases simply wouldn't be possible - hence why you can create different keys with different permissions.

      IAM is extremely flexible, unfortunately it cant protect from stupidity.

    3. Destroy All Monsters Silver badge

      Re: This is the kind of thing that keeps me up at night

      I also want to set limits on what and how much an AWS key can be used for.

      1) Use a specific credit card

      2) Amazon tells you often to use three-factor auth for the root account

      3) Use separate IAM users with specific permissions for specific tasks (this takes some getting into and also queries on stackoverflow, but the interface is really nicely done)

      4) ???

      5) Protect!

    4. Peter 26

      Re: This is the kind of thing that keeps me up at night

      Absolutely AWS is the main fault here, it is open for abuse. The worst thing is this guy phoned AWS support and told them what happened, but they still let all these services be created overnight. Surely you have a big red button that support can hit that says this account has been compromised, don't allow anything else which costs to run. But no support tell him to clean up the system himself and their "block" didn't actually work.

      I'd refuse to pay Amazon saying they were negligent.

    5. Nick Stallman

      Re: This is the kind of thing that keeps me up at night

      Did you miss the bit where they did contact him? Extremely quickly?

      And did you miss the IAM section which lets you specify very fine grained controls over your access keys?

      So....everything is fine then? You can sleep now.

  10. Destroy All Monsters Silver badge
    Windows

    This is like, your programming man.

    Dude, you are getting some advice:

    1) Credentials always in config files in a separate project not under version control. Config file reading is easy and can be done in 5 lines or less. Use XML, atttribute = value, whatever.

    2) When needed, slurp the credential files

    1. werdsmith Silver badge

      Re: This is like, your programming man.

      Yes, and live deploy the config file with the credentials in plain text in the same folder as your app binaries.

  11. Joe Harrison

    What is wrong with Amazon?

    I wanted to sign up for their free trial on day one but have never done so precisely because

    (a) I don't know what I'm doing until I have learnt

    (b) don't want my lack of knowledge somehow to trigger a massive credit card bill.

    You would think they would understand this.

    1. Destroy All Monsters Silver badge
      Windows

      Re: What is wrong with Amazon?

      Can't happen. You can see the bill develop hour by hour on the landing page, and it's not like you can accidentally the whole Virginia data centre.

    2. Anonymous Coward
      Anonymous Coward

      Re: What is wrong with Amazon?

      1. Use MFA with your primary account.

      2. Never create an IAM key with full account access.

    3. Anonymous Coward
      Anonymous Coward

      Re: What is wrong with Amazon?

      What is it that Amazon should understand? That you are technophobe?

      The guy who posted his keys to a github repository, private or not, is at fault. Keys are really just a better way of doing passwords and you would never post you'r password to your online banking anywhere either.

      Would you?

      Or if you would you really _do_ deserve what is coming ....

      1. werdsmith Silver badge

        Re: What is wrong with Amazon?

        The guy was only wanting a trial, and admitted his own shortcomings in points (a) and (b).

  12. paladin12

    Nominative Determinism

    Phil Haack? Really?

    1. dogged

      Re: Nominative Determinism

      yes, really.

      If you don't know who he is, you don't develop with Git and you're probably not a developer at all.

      1. Destroy All Monsters Silver badge
        Paris Hilton

        Re: Nominative Determinism

        But dogged, why should "not knowing who Phil Haack is" imply "nodev"?

        I use git btw.

      2. lurker

        Re: Nominative Determinism

        "If you don't know who he is, you don't develop with Git and you're probably not a developer at all."

        Really? I know who Donald Knuth is, and who Kernighan, Ritchie (RIP) and Ken Thompson are, but I'm not a developer because I don't know some (googled this) .NET blogger?

        I'm sure he may be a good programmer and a leading light in your personal firmament, but that's a pretty narrow definition of what a developer is.

        1. g e

          Re: Nominative Determinism

          Does that mean if we don't know who Alan Cox is we shouldn't use (say) filesystems or maybe linux?

        2. Loyal Commenter Silver badge

          Re: Nominative Determinism

          Indeed. I own a copy of Knuth, I've even read (parts of) it. I have a copy of The Camel that is so tattered that it'll probably fall apart if I picked it up (although I've not done so in what is probably over a decade), K&R is on the bookshelf somewhere. I cut my teeth programming BASIC on an Amstrad CPC 464 in the eighties, and taught myself to code in Z80 assembly when an early teen, but hell, I'm not a developer because despite coding in C# (amongst other languages), I've not heard of some blogger. I'd better run and tell my boss that he should cut my pay!

          Ironically, I could think of at least one (former) employee of the place I work at who almost certainly would have heard of this guy. This is also the person who likes to pick up any new technology and run with it, whether tried and tested or not, and leaves everyone else with the headache of having to learn it properly to fix his mistakes. A little less time reading blogs, and a little more obtaining a deeper understanding tends to yield more solid results...

          1. amanfromarse

            Re: Nominative Determinism

            I could have written your first paragraph about myself, but the second, no.

            I used to read several .net blogs, Hanselman, Haack, Conery, Skeet. You really think that they all chase the new shiny? My experience is the opposite of yours and that usually, not always, developers who didn't read .net blogs were 9-5ers.

        3. Anonymous Coward
          Anonymous Coward

          Re: Nominative Determinism

          >I don't know some (googled this) .NET blogger

          The only .NET blog that sticks out in my mind is Coding Horror and only because the guy is brutally honest about Microsoft's mistake and missteps up front.

  13. BrendHart

    I don't really trust those folks at Github.

    I am one of the poor sods who use their Github for Windows client which is indescribably buggy and every second release does something it shouldn't. Each time the software updates itself secretly in the background you just know that you are going to go on an adventure.

    They seriously need to do an audit of all their development processes.

    1. This post has been deleted by its author

    2. Destroy All Monsters Silver badge
      Paris Hilton

      Re: I don't really trust those folks at Github.

      Github for Windows client

      Why does such a thing exist??

      The point of github is that you can use

      1) a browser

      2) bog-standard git (which has its own problems and looks like a set of states, all different, arranged in a maze, but that is another discussion)

  14. Pascal Monett Silver badge

    "GitHub [..] has apologized for the error in its code"

    A proper apology would be to pay the bill. Anything else is just words.

    1. Bronek Kozicki

      Re: "GitHub [..] has apologized for the error in its code"

      I am not sure about that. Yes, it would be a nice gesture, but basically the root cause of the problem was 1) developer doing something stupid (storing authentication keys with code) with 2) free tools provided by MS and GitHub. In these circumstances it's really difficult to expect GitHub to pay up.

      It would be nice if they did, though.

      1. Pascal Monett Silver badge

        The developer didn't do something entirely stupid, he thought he was submitting to his private repository and, without GitHub's mistake, that's what would have happened and we would never have heard of it.

        He apparently would not have included the keys if he intended to use a public repository.

        To me this story is just another confirmation of "never put confidential data in the cloud". Ever. Under no circumstance.

        1. Anonymous Coward
          Anonymous Coward

          Do dumpster divers call the trash in front of the AT&T building "the cloud"?

          I think it is rather another confirmation of "double check what you are doing" and "defense in-depth is good".

          The fact that "the cloud" is involved is just an occasion to rightously sound off in the comment section, but has nothing to do with the poodle's kernel.

        2. Nelbert Noggins

          Except he did do something stupid, he had keys with the ability to spin up servers in his source. It doesn't matter whether it's a public or private repo on github, he handed the security of his keys to an external party who has no liability if they are abused.

          This is after the very public announcements and warnings from both GitHub and Amazon about storing keys in code.

          If the keys are for his application to do something he should either be using temporary tokens, IAM roles, or a restricted IAM account and if necessary pulling the values in from a config file/runtime insertion not storing them in code.

          Given how often sites are hacked these days and both companies specifically warn not to store keys in your source, not to mention that AWS provide alternative programmatic ways and examples to use their services, it doesn't really matter if you are using a public or private repo... there is no excuse other than sloppiness or ignorance for storing keys in the repository with the code.

    2. Robert Carnegie Silver badge

      Re: "GitHub [..] has apologized for the error in its code"

      I assume that the software licence says that they aren't liable for anything that happens. Inexcusable or not. These are the conditions that we accept.

      Having said that, I don't understand the technology, but it seems that it would be a good idea for the function that goes "Upload the project to the repository" to have a feature that goes "Don't upload that part of the project to the repository".

      As long as that worked, of course.

      1. Bronek Kozicki

        Re: "GitHub [..] has apologized for the error in its code"

        @Robert Carnegie that's what .gitignore file does, had he bothered to create/update it.

  15. Dexter
    Devil

    GitHub is the spawn of the devil.

    I don't get it. Why would you want to put all your source code out in the cloud where everyone can steal it?

    1. Thecowking

      Re: GitHub is the spawn of the devil.

      Because you want a repo for an open source project.

      1. Loyal Commenter Silver badge

        Re: GitHub is the spawn of the devil.

        'open source private repository' is something of an oxymoron though.

        1. asdf

          Re: GitHub is the spawn of the devil.

          More of cathedral than a bazaar.

  16. Anonymous Coward
    Anonymous Coward

    .gitignore

    Why was he hardcoding keys into the code anyway?

    Even GitHub itself has a guide explaining why you should stick keys into a config file and make sure it's in .gitignore...

    1. Anonymous Coward
      Anonymous Coward

      Re: .gitignore

      Because its a *private* repository he was supposed to be uploading to. In private repositories we do things like store keys and binaries because we abuse Github for non-build purposes, like the ability to git pull remotely and have a complete working copy of our code. Despite Github not being a backup service, our own private repositories where we work contain private keys, license files, database backups, all sorts of dll binaries. Convenience, ne? Until something like this happens.

      1. This post has been deleted by its author

        1. asdf

          Re: .gitignore

          @Pascal exactly well said.

        2. Anonymous Coward
          Anonymous Coward

          Re: .gitignore

          Which is that same thing as saying not to use any cloud, co location hosting, or possibly even the internet.

          Github is over SSL, and is stored encrypted at the remote end.

          If the story had been Dropbox spewed man's financial credentials in bad app update, the story would be focused on the tool and service, not how stupid the user was for using the service. The user had an expectation of security AND privacy.

          A private github *should be* no different to using any 3rd party cloud provider.

  17. Cool Hand Luke

    I am not a developer...

    ...so, I've never used Github. To me it sounds like a place where a bunch of like minded people meet up to discuss how much of a bastard they can be that day.

    1. phuzz Silver badge
      Thumb Up

      Re: I am not a developer...

      I think you'll find that the places for bastards to meet up and chat is in the comments thread of a BOFH article.

    2. Loyal Commenter Silver badge

      Re: I am not a developer...

      You have that confused with any bar in or near the Square Mile of the City of London...

    3. Robert Carnegie Silver badge

      Re: I am not a developer...

      I think it doesn't mean http://dictionary.reference.com/browse/git

      but maybe "Generous with Information Technology"

      On this occasion, too generous, to the wrong people.

  18. Anonymous Coward
    Anonymous Coward

    Had similar happen to me but to the tune of £12k!

    When AWS was first launched the keys were not able to be limited. Amazon soon changed that and now don't let you even get your root key anymore other then when first created.

    A company I previously worked for had code that had my keys (as a favour as the start-up company was run by a friend and they were using my S3 until they had set up a company payment card). - I know, stupid me, never share your keys with anyone, blah blah. I also share my car keys (oh-er missus) with friends and they do likewise and I didn't think the value of my amazon account was on par with the value of my car - after all I have my credit card linked as teh payment mechinism and that has a low limit, so whats the worst that could happen right?

    Well, new dev decided to put the companies jewels into GitHub (rather than the in house SVN I set up for them). And, as they didn't have a company payment card just used a public repo instead. First thing I knew was an email from Amazon on a Sunday morning saying my account looked compromised.

    It took me all of half an hour to lock everything down but as this had started sometime the night before my final bill was over £12k.

    One phone call to Amazon however took care of it all, they had seen this happen before and with barely any checks asked me for my estimated 'legit' usage and issued a credit against the account for the difference.

    As much as everyone seems to hate Amazon for it's customer focus (at the expense of employee focus by all accounts) it served me very well in this instance.

    Did make me wonder how much bit-coin someone managed to mine in that time.

  19. mark jacobs
    FAIL

    For source code, only use home-spun software (after all, you are a developer) and always put it on a VPN, and never a public server. If I want to see my own source code and I'm in Lesotho, I have to RDP to my VPN first. Without that first step, I can't get near it. Git is a bad idea and AWS is a toy. Don't trust third parties with commercial interests. Trust third parties with technical interests.

    1. asdf

      Can agree with most of what you said but the AWS is a toy ruined your whole post. I am not big on it personally either but I think it would blow your mind how much of the code S&P 500 companies rely on runs on AWS.

    2. Destroy All Monsters Silver badge
      Thumb Down

      AWS is a toy

      Pfff... hahahaha!

      Don't trust third parties with commercial interests.

      Go back to your Lesotho freedom/nocommercial hacker cellar and stay there.

  20. Gene Cash Silver badge

    Hardcoded passwords

    So he works for Cisco or Belkin then?

    This is his 100% his fault, and in this case the school of hard knocks is charging tuition.

    So five years down the road when the source has been lost (I know his type) then the app will have an unchangeable password, and they WON'T be able to change their Amazon compute services password.

  21. Anonymous Coward
    Anonymous Coward

    ..with a little help from Microsoft

    well, there's your root cause.

  22. sysconfig

    Sensitive data...

    ...doesn't belong in any repository. It lives outside the code tree (or, as a second best option, is flagged properly to be ignored in commits). Anything else is ignorance, especially when the repo is hosted in the "cloud". There is no such thing as private in the cloud. Fool.

  23. Someone Else Silver badge
    Coat

    Like to fell off my chair laughing...

    GitHub team member Phil Haack added, [...]

    Seems appropriate, somehow....

  24. Anonymous Coward
    Anonymous Coward

    Hahahahaha

    Its about time someone managed to find a way to get money out of a Capetonian.

    Its like getting blood from a stone usually.

    Capetonian Avarice is obscene.

    I got my haircut out there and the lady asked me for the equivalent of 2 pounds at the time. I gave her the equivalent of a fiver because I thought 2 quid was a bit unfair even for SA.

    Word got round and all the hard up millionaires (pounds not rands) got angry in case she put up her price. Legitimately angry.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like