back to article Typo in case-sensitive variable name cooked Google's cloud

Google has admitted that incorrectly typing the name of a case-sensitive variable cooked its cloud. Users of the Alphabet subsidiary's Google Container Engine customers “could not create external load balancers for their services for a duration of 21 hours and 38 min” on December 8th and 9th. The mess started when Google …

  1. ben edwards

    The real question is why would case sensitivity take down anything when it's not a password.

    1. Jos V

      Be careful now... You might be in line for some flaming, unless everybody stays nice and explains that anything other than MS file system is case-sensitive, accepting that not everyone knows everything. Hey-ho. I'm not down voting you.

      1. big_D Silver badge

        It was an enum in the Quellcode, it has nothing to do with the filesystem.

      2. This post has been deleted by its author

    2. jake Silver badge

      @ ben edwards

      "The real question is why would case sensitivity take down anything when it's not a password."

      The answer is that real operating systems (as opposed to so-called operating systems which are based on program loaders) are case sensitive everywhere. Kids trained on program loaders trying to program on real operating systems are quite prone to this kind of mistake. I've been seeing it for about 40 years ...

      1. LDS Silver badge

        Re: @ ben edwards

        It's not "real" operating systems or compilers, it's just old, outdated ones, which unluckily still exists today because even in IT there are too many people who believe "traditions" are more important than "reason".

        It would be really time that *nixes worshipper understand the '70s are long gone and their design is truly outdated now and it's time to redesign for the XXI century. But of course you can't discuss with someone who believe an OS is a kind of religion belief...

        1. Anonymous Coward
          Anonymous Coward

          @L D S

          What, a bit like someone using Roman numerals in the 21st century?

          1. LDS Silver badge

            Re: @L D S

            Look XXI, xxi, 21, twenty-first, Twenty-First have the same meaning, and you can easily understand it regardless of how it is written and displayed.

            Why? Because they are only graphic conventions... having a computer using graphic conventions for comparison instead of meanings is utterly stupid. Understandable when there were just a few kilobytes for data and code (and may, as an exception in such systems still existing today), no longer today.

            Otherwise why any sensible code does transform any numeral graphic representation to its internal representation which has a unique meaning, but doesn't for text?

            1. big_D Silver badge

              Re: @L D S

              On the other hand, in German Lesen and lesen are two completely different things and you cannot use a lower case l on the first one and an upper case on the latter, it would mess up the sentence and it wouldn't make sense, you would have to go back and read it several times - well there are examples in English as well, such as the misuse of affect and effect causing people to re-read sentences to get the meaning, because the sentence as written doesn't make sense.

              In this case Lesen mean read, reading, sensing, whereas lesen is the verb to read.

        2. Adam 1 Silver badge

          Re: @ ben edwards

          In programming, case is also by convention frequently used to indicate scope. You will have a public int VoteCount with a backing private field voteCount. Also used to make multi-word names readable.

          I have worked heavily with languages that are case insensitive (eg. Delphi) and those which are case sensitive (eg. c#). There are pros and cons to both approaches. The more important thing is having a tool that polices the convention and fix the case for you (eg. Resharper).

          1. LDS Silver badge

            Re: @ ben edwards

            I would fire any developers of mine attempting to use such a silly scheme. Infix caps is good, as long as it doesn't create subtle bugs because of case sensitivity.

          2. jake Silver badge

            Re: @ ben edwards

            "The more important thing is having a tool that polices the convention and fix the case for you"

            Oh. My. Fscking. Gawd/ess.

            So-called PROGRAMMERS want into the nanny-state these days?

            Just shoot me. The world is toast. We are done as a species. It's over.

            1. sabroni Silver badge
              WTF?

              Re: Just shoot me. The world is toast. We are done as a species. It's over.

              Because some people like an IDE that corrects casing mistakes? Computers are tools for humans, not the other way around. Grow up.

              1. sabroni Silver badge

                Re: A single misplaced one or zero can and will play all kinds of havoc on any given system.

                That's an argument for making your system NOT care if a human accidentally gets the case wrong. It's tricky enough to get computers to do what we want, why add extra difficulty by having "sessionAffinity" be a different thing to "SessionAffinity"? They both look like the session affinity to me.

        3. NinjasFTW

          Re: @ ben edwards

          Yes, let's just throw away the correct way of doing things because people today are too lazy to bother to learn how to do things correctly.

          After we get rid of the scourge that is case sensitivity, lets target other punctuation marks next. The semicolon is surely the work of the devil anyway.

          I'm not too fond of vowels either, 95% of words will still make sense without them and think of how much keyboard space we could save wtht vwls n thm!

          1. LDS Silver badge

            Re: @ ben edwards

            No one advocates sloppy way of writing. But meaning is the same regardless of a missing cap. Or do you mean you would discard a text only because a missing cap or comma? Do you mean if I copy someone else work just chaning the case sensitivity or fogetting a couple of commas is not a copyright infringment? I can copy the Linux kernel, change the case of each and every identifier, and republish it as my own work bypassing the GPL? <G>

            1. A Nother Handle
              Facepalm

              @ LDS Re: @ ben edwards

              "meaning is the same regardless of a missing cap."

              Is it really?

              Helping your uncle Jack off a horse.

              Helping your uncle jack off a horse.

              1. BongoJoe

                Re: @ LDS @ ben edwards

                Helping your uncle Jack off a horse.

                Helping your uncle jack off a horse.

                Why would I want my uncle shoot a horse in the first instance?

          2. RAMChYLD

            Re: @ ben edwards

            Ah, so you're a Hungarian style programmer!

            1. ben edwards

              Re: @ ben edwards

              Guys,

              When you're calling variables "getSome" and "getsome", and their content is spectacularly similar, how is the compiler or another developer supposed to know if you made a mistake? This is what I was hinting at in my original post, not some drawn-out argument over which OS is best or character-sets.

        4. jake Silver badge

          @ LDS (was:Re: @ ben edwards)

          "their design is truly outdated now and it's time to redesign for the XXI century."

          OK. I'll bite. Care to come up with a way to make an "A" blearily[0] compatible with an "a", and so-on with the rest of the alphabet & lower 8 bits? To say nothing of those of us who actually grok assembler?

          [0] I typoed "clearly", the splell checquer decided the above was a good option ... decided to leave it for punctuation ... not that I think the existing audience will understand the humor ;-)

        5. Ken Moorhouse Silver badge

          Re: XXI century

          >It would be really time that *nixes worshipper understand the '70s are long gone and their design is truly outdated now and it's time to redesign for the XXI century.<

          There's a touch of irony in that sentence. Talking about moving on from the 70's and then using Roman numerals to point to the future.

          Unix is a solid bedrock to base designs upon, as opposed to replacing it.

      2. david 12 Bronze badge

        Re: @ ben edwards

        :"Kids ... are quite prone to this kind of mistake. I've been seeing it for about 40 years"

        And and it was an obvious mistake even 40 years ago.

        But that was also a time when being able to print in different fonts was so new and exciting that it seemed like a good idea to do it everywhere.

    3. Ken Hagan Gold badge

      I think you are perfectly aware of the answer. Over half a century ago, when character sets finally became large enough to include two cases of letters, a native English speaker decided that centuries of common practice should be swept aside and case should be considered significant, because a case-sensitive search is just comparing bytes whereas a case-insensitive search requires more effort.

      Time has demonstrated that this was short-sighted. Computer scientists had no trouble adapting to the new rules and occasionally bent them into a feature. However, as soon as computers were picked up by the general public, problems started, and for those who need the assistance of a screen reader case-sensitivity is the work of the devil.

      Sorry to say it, but not very sorry, but BASIC and DOS got it right.

      1. Adam 1 Silver badge

        > just comparing bytes whereas a case-insensitive search requires more effort

        I wouldn't call doing a binary OR with 00100000 before comparing too onerous. (We are talking half a century ago before we needed to worry about whether the ô or ö are equal)

        1. alain williams Silver badge

          binary OR with 00100000

          Well, that might have worked in times past, but now non ASCII characters are being used - case folding is more complicated. Think Unicode.

          1. Adam 1 Silver badge

            Re: binary OR with 00100000

            > but now non ASCII characters are being used - case folding is more complicated

            You don't get complaints from me on that point, but my comment was addressing the statement "because a case-sensitive search is just comparing bytes whereas a case-insensitive search requires more effort" in the context of "Over half a century ago, " Unicode wasn't designed until the late 80s so certainly wasn't considered part of the effort when this decision was made.

            Unicode not only makes comparisons fun, but some collations treat some accents as equal for comparison but different for sorting. There is a world of hurt in there for English speakers.

        2. LDS Silver badge

          Most languages are more complex than English... many have collation rules that are a bit more complex and require a collation table. It's just a lookup, but it's something more complex than an OR and CMP.

        3. Ken Hagan Gold badge

          @Adam 1

          An OR with #40 would not have worked even 50 years ago unless you have already also tested that the character is in the range a-z.

        4. Kevin McMurtrie Silver badge

          It's not an equality test that is involved. Headers, configuration parameters, and interpreter variable names are looked up by with a fast indexing scheme, like a hash. For that to work, all parameters must always be case normalized before use. It seems trivial but it can cause an even more trivial key-value lookup to take 1000 times longer.

      2. jake Silver badge

        "Sorry to say it, but not very sorry, but BASIC and DOS got it right."

        No! They did NOT! Humans might be analogue in nature, but (most modern) computers are binary, by definition. A single misplaced one or zero can and will play all kinds of havoc on any given system.

        1. LDS Silver badge

          It's up to the human-computer intreface to translate across representation - and I do expect the computer to perform such a task and adapt to my nature, because it's my "slave", not viceversa, although many *nixes worshippers look to adore to be slave of their computers and being forced to adapt to them... it's a form of masochism, I believe.

      3. Anonymous Coward
        Anonymous Coward

        > Time has demonstrated that this was short-sighted

        Bullshit.

        1. Two things are the same if they are the same. If they are not the same, they are different. Give me this simple definition of "same" any day.

        2. In the advent of multi-lingual character sets, it is now extremely difficult to say that two different things are the same except for being in different case. ü and Ü is a simple example, but it's not just a case of having a huge lookup table for each character. From the Unicode FAQ:

        http://unicode.org/faq/casemap_charprop.html

        "Q: Is all of the Unicode case mapping information in UnicodeData.txt?

        A: No. <snip long explanation>"

        3. Trying using an application which provides case-insensitive matching by default (mysql for example). You can end up with all sorts of nasty edge cases.

        4. Case-insensitive processing breaks things like handling base64-encoded data. Have a look for example at Salesforce object IDs, which are 15 alphanumeric characters. Because people were loading these into Excel which does case-insensitive lookups, they were forced to add an extra three characters which encode the upper/lower case state of the previous 15 characters.

        https://help.salesforce.com/apex/HTViewSolution?id=000004383&language=en_US

        https://astadiaemea.wordpress.com/2010/06/21/15-or-18-character-ids-in-salesforce-com-%E2%80%93-do-you-know-how-useful-unique-ids-are-to-your-development-effort/

        http://www.billgreenhaw.com/2014/02/27/difference-between-15-and-18-character-ids/

    4. phuzz Silver badge
      Thumb Up

      Let's just admit it, the Amiga had the best take on case sensitivity.

      It would allow you to capitalise files however you wanted (eg File1 file2 FILE3), but would search in a case insensitive way unless there was two identically named files in the same directory (so File1 and file1 are different files).

      Of course, only a crazy person would start creating File1, file1, FILE1 etc.

    5. thames

      What does "case" mean in languages that don't use a European alphabet? Even defining what a "letter" is can be subject to many complex rules in some alphabets, sometimes with no clear answer. The only rational way of dealing with the issue is to talk in terms of unicode code points, or for OS internals, streams of bytes.

      Case insensitivity came from the days when teletypes and punch cards formed the user interfaces, and they typically couldn't deal with case. Case insensitivity wasn't created because it was "better". It was an ugly hack that was done to get around short term hardware issues. Now think about a world in which believe it or not, not everyone speaks English or has a name that is spelled like "Smith" or "Jones".

      Some operating systems and programming languages have huge case insensitivity related legacy issues dating from the punch card and teletype days. Those legacy issues form an interesting historical quirk, but they will eventually disappear as those legacy case insensitive operating systems and languages die out.

    6. razorfishsl

      Obviously you never worked with java....

  2. Andrew Jones 2

    "Bootnote: Yes, we are aware El Reg is not always a paragon of purity, and that there is probably a pot-kettle-black mistake in this very story."

    To my knowledge a) you aren't charging people to have access to your backend server(s) and associated hardware and b) this is either their 3rd or 4th major TITSUP event this year is it not?

    1. graeme leggett

      And humans can route round errors (and regional differences) in spelling, or question them.

    2. Arctic fox

      @Andrew Jones 2 Re:"To my knowledge "

      Agreed. Furthermore, a typo committed by one of the El Reg journos would in all likelihood not cause this site to go offline for most of 24hrs!

      1. dotdavid
        Headmaster

        Re: @Andrew Jones 2 To my knowledge "

        "a typo committed by one of the El Reg journos would in all likelihood not cause this site to go offline for most of 24hrs"

        Oh I dunno, a particularly grievous grammatical error might conceivably cause a spike in the commentard count that could bring the site to its knees...

        1. Arctic fox
          Thumb Up

          @dotdavid "Oh I dunno, a particularly grievous grammatical error might conceivably......."

          See icon.

  3. MacroRodent Silver badge
    FAIL

    1

    So the whole Google cloud was brought down by literally a 1-bit error! (the difference between the encoding of lower and uppercase ASCII Ietters is in one bit).

    1. Anonymous Coward
      Anonymous Coward

      Re: 1

      But for another part of google "bytes are big"

      http://m.theregister.co.uk/2015/12/11/extra_stingy_google_cloaks_android_in_red_screen_of_smallmalwaresmall_dearth/

      Just not big enough to spot.

  4. frank ly Silver badge

    Why wasn't this noticed by the compiler ....

    .... or whatever other form of code processing and loading?

    1. getHandle

      Re: Why wasn't this noticed by the compiler ....

      Don't be silly - modern, hip, cloudy operations don't use anything so archaic as compiled languages! Or reviews, QA, testing, etc, etc...

    2. Steve Gill

      Re: Why wasn't this noticed by the compiler ....

      Even if it was compiled the compilation might have been completed before the real error occurred - the change to case sensitivity in the API. Everything coded prior to that point would have to be retested (and obviously wasn't). That should have been a red-flag kind of change (not “a minor update to the Compute Engine API”) - alert everyone and make sure you have a pro team on standby for the implementation.

  5. Ken Moorhouse Silver badge

    Holy Shift

    The new fashionable expletive of choice.

  6. Steve Button

    It's not about "proof reading"

    It's about test coverage. You are thinking of 20 years ago.

    1. Scott 53
      Headmaster

      Re: It's not about "proof reading"

      proofreading

  7. Michael Thibault

    There may be ...

    an omnipotent omnipresent omniscient invisible entity who gets bored from time to time and decides to fuck with people's heads. I could be wrong, but ...

    Alphabet. Shift. Oops!

    Nyuk! Pfnarr!

    That was a good one, a tight little noose.

  8. Loud Speaker

    Obviously ...

    Just as I misread it (insufficient coffee)

    Someone pressed the "shit" key instead of the shift key!

  9. batfastad

    camels

    googleCloudBittenByCamelCaseSnafu?

  10. petur

    irony

    To make such a mistake in a company called 'Alphabet'

    1. David Lewis 2
      Trollface

      Re: irony

      Does that mean I can create a company called "alphabet" or "ALPHABET" or "AlPhAbEt" & I won't get swamped by their lawyers as they are "obviously different"?

  11. Quortney Fortensplibe
    Mushroom

    Do No Evil?

    "..this is not the level of quality and reliability we strive to offer you, and we have taken and are taking immediate steps to improve the platform’s performance and availability..."

    That reads like they took the guilty party out the back and shot him and are now hunting down his known associates.

    1. Mark 85 Silver badge

      Re: Do No Evil?

      Nah.. they just broke a few fingers. The screams of pain and him/her walking around the office wearing a cast got the point across.

  12. Forget It
    Holmes

    The world is everything that is the case.

    Tractatus Logico-Philosophicus Wittgenstein

    http://genius.com/2201817

  13. thames

    Just Had a Look

    I can't readily find where the bug occurred, as none of the reports I've seen are explicit about that. However, I downloaded the Kubernetes code to see what it's like (that's one of the nice things about open source).

    It's all written in Go. I'm not that familiar with Go, but a quick grep through the code for sessionAffinity turns up lots of things like the following:

    info.sessionAffinityType = service.Spec.SessionAffinity

    There are lots of use of "sessionAffinity" in Go, JSON, and YAML, and HTML. It has different case in different applications - both "sessionAffinity" and "SessionAffinity". It's easy to see how someone got mixed up.

    I'm not a big fan of camel case or really long names. Names should be short, simple, and distinctive, and people should either use all lower case or capitalise the beginnings of all variables and labels, i.e. "SessionAffinity" rather than "sessionAffinity". Having both styles of spelling in the same project (keeping in mind we are talking about more than just Go files) is asking for trouble.

    The real screw-up however in my opinion is the fact that this wasn't covered by testing. This may have been due to an oversight in test construction, but it's possible that testing simply isn't complete enough.

    A quick line count with "wc" shows 700,771 lines of Go, of which 30,867 is in the test directory. Or to put it another way, that's roughly 22 lines of application code for every line of testing code. To put that in perspective, I have a project that has roughly 30,000 lines of 'C' code that is being tested by more than 1,000,000 lines of Python, for a ratio of 33 lines of test code for each line of application code (in other words, the reverse ratio), and I can't see a reasonable way of reducing that while being confident in the results.

    The worst thing to see people say is "the compiler or IDE should catch the errors". No, in any non-trivial project you cannot rely on the compiler to find errors. You can't say "it compiles, ship it!" The compiler may compile it, but to know if it actually works you have to test it.

    So the question I have for Google is why wasn't this caught by testing?

    1. sabroni Silver badge

      Re: It's easy to see how someone got mixed up.

      It is. Treating "sessionAffinity" as being the same as "SessionAffinity" would've fixed that. You'd have to find another way of differentiating the two if they are actually different, but adding another word to indicate the difference would help. I quite like long variable names as long as each word contributes some meaning.

      Remember back in the day when we used prefixes to indicate types of things? Can anyone remind me why that was such a bad idea?

      1. thames

        Re: It's easy to see how someone got mixed up.

        "Treating "sessionAffinity" as being the same as "SessionAffinity" would've fixed that."

        Actually, it wouldn't, because this is just one example of a misspell. It could just as well have been "sessionAffiinity". You've only solved a very narrow problem without really solving the problem in general. The only real solution is testing to see if something actually works.

        "Remember back in the day when we used prefixes to indicate types of things? Can anyone remind me why that was such a bad idea?"

        It was a bad idea because it added a lot of cryptic prefixes for no real good reason. If you changed the type, you had to change the variable or constant names. And what about when you defined your own types? Do you create your own prefixes? And what about when everything became an object? Do you prefix almost every variable with something that means "object". How was that supposed to be useful? And how would it work with dynamic languages where the base type of the reference can potentially change?

        It really wasn't very useful. That's why it never really caught on outside of a fairly narrow niche of developers.

  14. Spaceman Spiff

    Code review?

    Even with proper code review procedures, this sort of stuff can be missed since most of the reviewers will not perform close inspection during the process. Sigh. That is just how it is. Better static analysis tools might have caught this before the review process started.

  15. Anonymous Coward
    Anonymous Coward

    Why did existing tests not catch this?

    Why did existing tests not catch this?

    I can understand that if it is legacy code it my not have test coverage, but even then if you are adding/changing code you would (whether TDD or post-dev-work) add tests accordingly.

  16. Dick Pountain

    Camel-case makes for more readable naming conventions in large programs - eg. significant lower-case prefixes like eFoo, eBar, eZot, sFoo, sBar, sZot. That's a separate issue from whether or not to enforce such distinctions with case-sensitive compilation: there are strong arguments for and against. Either way, a smart IDE ought to warn about variable accesses that differ only in case.

  17. Borg.King

    So many opportunities to find this sooner

    1. Naming standards

    2. Test cases

    3. Test environment(s) / dog fooding

    3b. Post deployment verification checks

    4. Pre-production environment(s)

    4b. Post deployment verification checks

    5. Post Release verification checks

    (Ok, so 4b & 5 are likely exact dupes of the already failing 3b)

    Working here east of Seattle, I'm unhappy that my people are 'migrating' to Google, but hopefully they'll be performing better than the incumbents.

  18. Anonymous Coward
    Anonymous Coward

    strange business model?

    "Users of the Alphabet subsidiary's Google Container Engine customers ..."

    I'm wondering who would use their customers and for what.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019