back to article Thunderstruck: Azure Back in Black(out) after High Voltage causes Flick of the Switch

Microsoft is blaming bad weather for the massive outage that knocked a number of Azure cloud and Visual Studio Teams services offline Tuesday. The Windows giant revealed its South Central US facility in Texas was crippled after severe storms and lightning strikes overloaded its cooling equipment, forcing its servers and other …

  1. Chris Tierney

    Title should be

    Real cloud strikes fake cloud.

    1. Jack of Shadows Silver badge
      Mushroom

      Re: Title should be

      Given the part of the world, guess they finally ticked off Tlaloc.

    2. diodesign (Written by Reg staff) Silver badge

      Re: Title should be

      Sensing some resistance to our AC/DC puns. Get it, AC, DC. Works on many levels.

      (Also, think we've done real-fake clouds.)

      C.

      1. NanoMeter

        Re: Title should be

        Loved the AC/DC puns. Keep em coming.

      2. Hans Neeson-Bumpsadese Silver badge

        Re: Title should be

        Sensing some resistance to our AC/DC puns. Get it, AC, DC. Works on many levels.

        Excellent punning - mho please

      3. Alistair Silver badge
        Windows

        Re: Title should be

        @Diodesign

        Since we're going musical I guess it's safe to say that MS has seen the clouds from both sides now?

        1. Hans Neeson-Bumpsadese Silver badge

          Re: Title should be

          Since we're going musical I guess it's safe to say that MS has seen the clouds from both sides now?

          That musical reference is off-topic - (big yellow) taxi for you

  2. Anonymous Coward
    Anonymous Coward

    'This, in turn, lead to the problems that plagued Azure service users in Europe'

    Texas shook Europe: How many times has the absence of real redundancy put a 'black cloud' over the cloud? There's too much inter-dependency why?

    1. whitepines Silver badge
      Trollface

      Re: 'This, in turn, lead to the problems that plagued Azure service users in Europe'

      It's Microsoft. Did you expect proper design?

    2. Dan 55 Silver badge

      Re: 'This, in turn, lead to the problems that plagued Azure service users in Europe'

      There's too much inter-dependency why?

      Because it's Microsoft where everything depends on everything else. Did you miss the C:\Windows\System32 directory over the last two decades?

    3. LDS Silver badge

      Re: 'This, in turn, lead to the problems that plagued Azure service users in Europe'

      Is also putting a datacenter in areas with a good chance to be hit by sever weather each year a good idea? I understand those areas are also cheap, full of cheap workers, and with politicians willingly to subsidize anything that will bring in some jobs, but that's the result....

      1. Aladdin Sane Silver badge

        Re: 'This, in turn, lead to the problems that plagued Azure service users in Europe'

        Sever weather? Is that the kind that cuts things off?

      2. Jon 37

        Re: 'This, in turn, lead to the problems that plagued Azure service users in Europe'

        For certain uses, you want the datacenter near the users. There are plenty of users in south east USA, but the whole area is at risk from hurricanes. So putting a datacenter there is a perfectly reasonable decision, balancing the risks and benefits.

        Of course, for an organisation with multiple datacenters, designing your worldwide directory service to depend on any single datacenter is very silly.

        1. LDS Silver badge

          "For certain uses, you want the datacenter near the users."

          That's why a datacenter in Texas was serving European users? I understand Texas is Biiiiiiig, but it doesn't stretch across the Atlantic Ocean yet....

          1. Loud Speaker

            Re: "For certain uses, you want the datacenter near the users."

            Texas is Biiiiiiig, but it doesn't stretch across the Atlantic Ocean yet....

            Don't worry, Chuck Norris is working on that!

      3. Rastor728

        Re: 'This, in turn, lead to the problems that plagued Azure service users in Europe'

        "Is also putting a datacenter in areas with a good chance to be hit by sever weather each year a good idea?"

        ????? Every place can/might have extreme weather each year, and when it isn't weather it is earthquake, volcano, tsunami, stampedes, bugs (crawly type), war, crime etc...

        1. LDS Silver badge

          "Every place can/might have extreme weather each year"

          True, but not every place have an hurricane/tornado/typhoon/monsoon season... with a far bigger chance to be hit by extreme weather than others. Planning for earthquakes is another obvious thing to take care of - some areas have far larger chances than others. Tsunamis are usually avoided when you're not close to the sea.

          Sure, you can still be hit by an asteroid anywhere, or Trump cold declare war to anybody, but the chances are far lower. That's called risk-assessment, but if beancounters preempt any reasonable choices and their eyes are filled with tears when they see cheap workers and subsidies, your chances of a failure becomes far, far bigger.

  3. Anonymous Coward
    Anonymous Coward

    Zap!

    Massive electric discharges coming out of the sky! How is this possible? Who knew this could happen? How were we to know it could affect electrical equipment?

  4. Leedos

    50% restored (for me)

    I have 4 VM's in that data center. 1 survived, 1 was restored, still waiting on the other 2.

    Didn't Ben Franklin come up with something to prevent this kind of thing?

    1. TheRealRoland
      Coat

      Re: 50% restored (for me)

      Go fly a kite, will ya?

    2. Anonymous Coward
      Anonymous Coward

      Re: 50% restored (for me)

      So did you have to actually do any of this recovery yourself, or was it just sitting it out and waiting for your VMs to go green again?

  5. Claptrap314 Bronze badge
    Facepalm

    Not impressed

    Thunderstorms in Texas at first of September--whodathunk? Yeah, this is bad design at the datacenter level. And that data center appears to be a single point of failure for a lot of their "cloud".

    Go Microsoft!

    1. MyffyW Silver badge

      Region == Datacenter. WTF?

      Whilst I already knew some MS regions were represented by only one datacentre, it strikes me as odd this is the case in a highly developed part of the world that is known to have it's fair share of freak weather.

      As for one region being able to take down Azure Active Directory, Calrissian's conjecture would seem to apply - This deal is getting worse all the time.

    2. Version 1.0 Silver badge

      Re: Not impressed

      The weather here (I'm on the gulf coast) is actually quite mild this year, sure we've had a few thunderstorms but nothing exceptional - Texas just had a tropical depression, not even a storm.

  6. Anonymous Coward
    Stop

    Dirty deeds

    Or the useless riff raff in charge.

    People should pay a bit more for better hosting, as after all money talks...

    1. HieronymusBloggs Silver badge

      Re: Dirty deeds

      Upvote for multiple AC/DC references. I guess the downvoter is not a fan.

      1. J P
        Coat

        Re: Dirty deeds

        Presumably concerned that starting on AC/DC puns is just the first step on the highway to hell.

      2. Anonymous Coward
        Anonymous Coward

        Re: Dirty deeds

        Nah, I rubbished his £1000 iphone toy in a different thread. Had it been AC (DC), it would have been a different story.

  7. Destroy All Monsters Silver badge
    Trollface

    Those engineers.

    “Engineers have successfully restored power to the datacenter. Additionally, engineers have recovered a majority of the impacted network devices”

    Is it legal to use Moties groundside?

    1. Neil Barnes Silver badge

      Re: Those engineers.

      Engineers yes, watchmakers no.

  8. Kevin McMurtrie Silver badge

    The magic of the cloud

    Trust us, you don't need to know where your compute power is hosted. It's virtual, safe, and infinitely distributed. We swear it's not all ending up in one place.

  9. tempemeaty

    I wonder if it's a warning?

    What was it...a week or so ago Microsoft acted unprofessionally and threatened the growing social network Gab.ai with kicking them off Azure cloud service if they didn't censor two posts from two users of the social network? If Microsoft will pull the service out from under your business because they don't like one or two of your customers, that makes using Microsoft's Azure a unpredictable risk to any that might use them. Maybe God is finally warning Microsoft or Karma caught up with them...

  10. Hans 1 Silver badge
    WTF?

    Texas - Europe ?

    Ok, so, let's try to understand this ... MS claims your data is stored in Europe when here we have proof it is not ... and nobody picks it up ?

    Given MS' long and winding track record (recent AND past), who would trust MS to be able to implement a stable and disaster-safe infrastructure ? I mean, come on, with the resources they have ... everytime there are WIndows updates due, the infrastructure becomes unresponsive .... now, lightning in the US causes Europe's service to go TITSUP ?

    1. Lusty Silver badge

      Re: Texas - Europe ?

      Nope, this is fully documented. Azure AD has always been a global service where data is not guaranteed to be in region.

      Also, there are disaster recovery procedures in place which would have been used if recovery were not underway, which it was. What I think you mean is business continuity, and I agree it's disapointing that AAD isn't designed for availability across regions.

      1. Dan 55 Silver badge

        Re: Texas - Europe ?

        So how are you supposed to comply with GDPR if AD is sending PII all over the world?

        1. ArrZarr Silver badge
          Facepalm

          Re: Texas - Europe ?

          By not using it for PII?

          If MS say it's not compliant with GDPR then you don't use it for anything that's covered by same.

          1. Dan 55 Silver badge
            FAIL

            Re: Texas - Europe ?

            So in other words you can't populate the name, address, or contact information fields because of the way they've designed it.

            Perhaps it should be renamed to Azure pretty fucking useless directory.

            1. Aladdin Sane Silver badge
              Trollface

              Re: Texas - Europe ?

              No, Texas are from Scotland and Europe are from Sweden.

            2. Andy_Lee

              Re: Texas - Europe ?

              Nope because WORK related contact details are not to be considered as PII

              1. Alex Brett

                Re: Texas - Europe ?

                While business names and addresses are not PII, if you e.g. have your employee's names and contact numbers, that very much *is* PII...

                1. Anonymous Coward
                  Anonymous Coward

                  Re: Texas - Europe ?

                  "if you e.g. have your employee's names and contact numbers, that very much *is* PII..."

                  That's true, but if you have a business reason to store them, such as the ability to contact employees and ex employees for HR purposes then you're fine. Naturally you need to manage that info and have lifecycle processes, but GDPR was designed with these scenarios in mind. Sadly it's nearly 70 pages long so most people don't read it and instead make wild assumptions based on sales seminars.

                  1. Dan 55 Silver badge

                    Re: Texas - Europe ?

                    GDPR says has to be an "adequate level of data protection" for data held in non-EU countries. If AD data is replicated worldwide to many countries, that can't be guaranteed.

        2. Anonymous Coward
          Anonymous Coward

          Re: Texas - Europe ?

          @Dan55 step 1 would be learning GDPR rather than ranting bollocks on the Internet. AD isn't in scope for some exceptionally obvious reasons.

  11. Anne-Lise Pasch

    VisualStudio.com wont let me release anything. Despite all my activity/servers being in Europe. Things build fine. I guess the deployment agents are in the US. Or the controllers for them.

  12. Milton Silver badge

    How odd ...

    How odd ...

    ... that when modem'ing into the university mainframe in 1981, a lightning strike in Texas didn't stop me working.

    ... that when developing a C++/ASM module to support an airline Clipper application in 1993, the weather in America didn't bring everything down.

    ... that while working on an 800Mpx 5-layer image yesterday on my four-year-old desktop PC, some rain 4,000 miles away didn't bring me to my knees.

    The efficiency, robustness, reliability and security of 'cloud' is truly a wonderful thing. Until you find that you're paying for latency, sluggishness, mysterious interruptions, literally endless excuses and get-out clauses, and single points of failure arising from the inclusion of absurdly over-complicated and often unnecessary systems, all of which, when you come down to it, are primarily contrived to extract money from you, hold your data to ransom, entrap your business's livelihood, spy on you and steal your IP.

    Go right ahead, make yourself dependent on this or that monopolistic internet giant. Tell yourself they have your best interests at heart. Wait till you've foolishly let yourself become dependent, wriggling on the punji sticks of their 'ecosystem'. And when they put the prices up to whatever doesn't quite bankrupt you, squeal as loud as you like.

    Better still, make sure you grabbed that 'cost-saving' bonus last year and ran for the hills ...

    1. ArrZarr Silver badge

      Re: How odd ...

      Going back through all of industrial history, the trend has been centralising power in the name of efficiency - Enclosure of fields between 1500-1800, the great mills of the industrial revolution and now the cloud.

      You may as well complain that flooding in Bangladesh drives the price of RAM up when in the past you were able to cobble together some VRAM yourself with delay line memory or how flooding in central Europe causes a shortage of iceberg lettuce.

      Centralisation happens, efficiency improves, local effects at the point of centralisation become more important.

      But sure, rail against the cloud for the obvious WOMBAT that it is.

    2. Alistair Silver badge
      Windows

      Re: How odd ...

      @ Milton.

      I have an email to send to quite a few folks I've worked with over the last 20 years soon. I think that main paragraph, in white 6 pt text on a white background may just sneak in there......

  13. Anonymous South African Coward Silver badge
    Joke

    Wot? Don't they have Hyper-V replication or clustering in place for their customers' valuable VM's?

    1. Anonymous Coward
      Anonymous Coward

      "Don't they have Hyper-V replication or clustering in place for their customers' valuable VM's?"

      That depends on the customer's design choice, not on Microsoft.

      1. Olivier2553 Silver badge

        "That depends on the customer's design choice, not on Microsoft."

        Yes? But no. What would be the point of going to an external provider if you have to implement all the replication by yourself? The service the cloud is selling *is* the replication and high availability. Else it is a co-hosting, not a cloud.

    2. Dr Who

      Someone misread the design spec and implemented clusterfucking.

  14. Anonymous Coward
    Anonymous Coward

    One point twenty one gigawatts

    Great Scott! Didn't anyone see this coming?

    (Doc Brown voice)

    But seriously, who could have predicted a cloud service being impacted by a thunderstorm?

    Did the weather forecasters mess up again?

    (cough freak storm at DEFCON 18 /cough)

  15. Pseudonymous Howard

    Heavy Weather

    by Bruce Sterling. Always worth a (re)read.

  16. Anonymous South African Coward Silver badge

    The Day after Tomorrow...

    1. Aladdin Sane Silver badge

      Is a shit film.

  17. PeterM42
    Facepalm

    Our Datacentre is fully duplicated and..........

    .......NOTHING can go wrong,

    go wrong,

    go wrong,

    go wrong,

    go wrong,

    go wrong,

    go wrong,

    go wrong,

    Clouds are part of changeable weather. Why would anyone RELY on them???

  18. Ima Ballsy
    IT Angle

    I'm confused .....

    Isn't the WHOLE cloud thing thing suppose to be UPTIME of 99.99999 and your system runs all over the world. If one location goes down, then the others take over.

    Now tell me, isn't THAT suppose to be how it works ?

  19. StuntMisanthrope Bronze badge

    Shot down in flames!

    Give it a KiXtart. #bigballs

  20. Jay Lenovo Silver badge
    Devil

    Whole Lotta Rosie lights on my status panel

    The Azure Cloud shaken to the foundation,

    while customers got "The Jack".

    1. hopkinse

      Re: Whole Lotta Rosie lights on my status panel

      only those with big balls succeed :-)

  21. Hans Neeson-Bumpsadese Silver badge

    Cloud storage

    Don't know why,

    There's no SAN up in the sky

    Stormy weather......

  22. Anonymous Coward
    Anonymous Coward

    Issue with power supply - then use your own

    Power supply issue and surges risk your equipment.

    Option 1 - Switch to generator

    Option 2 - Power everything off

    & MS choose Option 2.

    Say no more...

  23. francis.mondia.et
    Childcatcher

    A Moment of Silence to fellow Sysadmins Sorting This Out

    And for the families too who will be missing family members for dinner, breakfast, lunch for a few more days..

  24. Claptrap314 Bronze badge

    By the numbers

    Lots of odd presumptions here. I'm going to try to sort some of them.

    1) Data plane != control plane. It is entirely possible (but really, really bad design) for the AD information to be held in European servers that won't work properly unless they are talking to some server in Texas.

    2) Data centers go down. That's the point. It is really, really dubious to claim that your system is running a cloud of any sort if a single datacenter going out can take it down. In SRE, you plan for planned outages & unplanned to happen at the same time. Anything less is NOT resilient in any meaningful way.

    Therefore, it is perfectly rational to put a datacenter in hurricane alley, or tornado alley, or right on top of the San Andres fault, or on the downwind side of a volcano. Datacenters are sited primarily for access to power and workers, and are engineered (in theory) to have physical uptime of 90-95%, at least, that what I saw.

    It's the wiring and coding between datacenters that is the magic of SRE that can give five nines. Not what the individual datacenters can do.

    3) Azure is hardly the most mature cloud offering out there. It's just silly to dump on the entire idea of cloud because a company with a history of poor reliability is having trouble when it starts a major undertaking.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019