back to article Microsoft Azure Europe embraced the other GDPR: Generally Down, Possibly Recovering

Microsoft Azure tumbled over in northern Europe – and services have effectively stayed down for unlucky customers for around five hours. The disruption was caused by problems within the cloud platform's storage and networking systems, we're told. Today, from 1744 UTC to now, at time of writing 2115 UTC, according to the …

Page:

  1. Anonymous Coward
    Anonymous Coward

    More Microsoft CloudFog

    Its been said many times before on here.... Cloud means someone else's computer. But you can't deny Cloud convenience is great when everything is working... However you've got to wonder, how many *if any*, senior executives jumping on the Cloud conveyor belt, planned for days like this...

    1. Anonymous Coward
      Anonymous Coward

      Re: More Microsoft CloudFog

      None of them - moved on to next gig by then most likely

      1. Steven Raith

        Re: More Microsoft CloudFog

        The plan was to be able to blame someone else.

        Not their servers, are they? So not their fault. Bonus please!

        Steven R

        1. Phil W

          Re: More Microsoft CloudFog

          "The plan was to be able to blame someone else."

          Yes that usually is the plan, and both IT management and senior management tend to be on board with the idea that not having any control or ability to fix it yourself during an outage is acceptable, right up until there actually is an outage at which point senior management shout at IT management to fix it and IT management shout at the people who actually know what they're doing to fix it, and we say "Sorry boss, but you took it out of our control".

    2. TheVogon Silver badge

      Re: More Microsoft CloudFog

      "Engineers have identified the root cause, and actively working to mitigate the issue. Our telemetry has shown improvement, and next update will be provided in 60 minutes or as events warrant."

      1. hplasm Silver badge
        Happy

        Re: More Microsoft CloudFog

        " next update will be provided in 60 minutes or not..."

    3. Scott Marshall

      Re: More Microsoft CloudFog

      They planned for it by creating BaaS - Blame As A Service.

    4. Fungus Bob Silver badge

      Re: More Microsoft CloudFog

      "you've got to wonder, how many *if any*, senior executives jumping on the Cloud conveyor belt, planned for..."

      Planned? You have some rather strange ideas about life in the corporate world...

  2. Anonymous Coward
    Anonymous Coward

    Cloudy, with a chance of outages

    What do you do when it rains in the cloud?

    1. Anonymous Coward
      Anonymous Coward

      Re: Cloudy, with a chance of outages

      Get pneumonia(?)

  3. Hans 1 Silver badge
    Coffee/keyboard

    Who can tell MS not to install updates on all servers in a tier at once ?

  4. Anonymous Coward
    Anonymous Coward

    MTBF vs Blast Radius

    AWS, GCP and Azure are all more reliable and more secure than a traditional data centre but that doesn't mean you don't need a DR plan. Well Architected cloud applications can survive failures.

    The difference with public cloud is if it has an outage its major news due to its blast radius, not usually true when one of your own DC's has an issue.

    1. Remy Redert

      Re: MTBF vs Blast Radius

      Pretty much. If you must run in the cloud, make sure you're spread across multiple cloud providers.

      Fortunately their platforms are all highly interopable to make this kind of safe arrangement much easier to achieve. Right?

      1. Jack of Shadows Silver badge

        Re: MTBF vs Blast Radius

        No, they aren't directly interoperable although there's a few players out there that can build across providers. However, if you are doing your design correctly, everything should be loosely-coupled which mitigates things a bit. The problem, as I see it, is that they're holding it wrong: the providers and the clients.

        1. AMBxx Silver badge

          Re: MTBF vs Blast Radius

          Azure gives you the option for geographic redundancy. Easy for storage and cold standby, I'm not sure how complex it becomes for VMs (not my thing really).

          Multiple providers not likely to be the solution - you're just adding complexity.

          1. Anonymous Coward
            Anonymous Coward

            Re: MTBF vs Blast Radius

            > Azure gives you the option for geographic redundancy.

            It does, but many cost benefits of moving to the cloud are wiped out once you add the redundancy options.

            Move to the cloud for any two (and possibly just one) of cost saving, scale on demand and redundancy.

          2. LeeE Silver badge

            Re: MTBF vs Blast Radius

            "Azure gives you the option for geographic redundancy."

            Compliance with local legislation, such as GDPR, may limit your options for geographic redundancy.

            1. AMBxx Silver badge

              Re: MTBF vs Blast Radius

              >> Compliance with local legislation, such as GDPR

              There are no end of centre to choose from. Even in the UK there are 3.

              1. Anonymous Coward
                Anonymous Coward

                Re: MTBF vs Blast Radius

                Not all services are available in all regions and some fundamentals may only delivered from the more mature regions but yes, you can deliver within the EEA if you do the work. I'd be reluctant to deploy in the UK until it becomes clear whether it can provide data adequacy post Q12019.

    2. johnnyblaze

      Re: MTBF vs Blast Radius

      I highly question that comment. Sure, cloud providers have distributed networks with umpteen layers of protection and resiliance, but when they go down, and they do very regularly, they go down spectacularly and take down thousands of customers. On prem solutions may seem 'old hat' now, but a well designed system can be rock solid and reliable, and I'd take a good one over cloud any day of the week.

    3. teknopaul Bronze badge

      Re: MTBF vs Blast Radius

      Where I work we have never had five hours of down time ever.

      1. PM from Hell
        Devil

        Re: MTBF vs Blast Radius

        =Thats the most idiotic thing I have ever seen anyone publish, Most of us will never go further than stating that "things are OK" or there has not been a major incident for "a while" for fear of offending the gods of hardware or the demons of database recovery. Good look with your weekend outage by the way & I hope you have the phone numbers for a 24 hour pizza delivery service.

  5. gerdesj Silver badge

    Title

    Wouldn't it be nice if threads that had a title with an OS in it had a no Anon rule?

    Has anyone really bothered to value their el Reg karma value?

    Does anyone really give a shit?

    1. Anomalous Cowturd
      Unhappy

      Re: Title

      Yes, no, and no.

      My silver karma badge fell off when I stopped posting regularly, quite some time ago. This site has gone downhill / changed in the past two years.

      I blame Sarah, for leaving.

      1. Inspector71
        Devil

        Re: Title

        I blame Eadon.

    2. ratfox Silver badge

      Re: Title

      Some employers know that their employees are posting on the Reg. The employees would like to freely comment, even when they disagree with their employer.

      1. Anonymous Coward
        Anonymous Coward

        Re: Title

        ... Which is why this Anon posts as such. 'plausible deniability'.

    3. Anonymous Coward
      Anonymous Coward

      Re: Title

      Has anyone really bothered to value their el Reg karma value?

      Well, I'm +50k up, 9k down. I reckon I can afford to piss people off, but I stay anon because I don't want badeg or name to distract from the contents. I know why some people are anon: it's either that or they would not be able to comment and given that such people are generally of a level that their input is valuable I prefer they stay anon.

      Besides, you know that any comment can be used against you when you cross the US border, right? You may have the right to silence, but that's pointless if you have already spoken online..

      1. Anonymous Coward
        Anonymous Coward

        Re: Title

        If you look at "my posts" you see your anonymous posts. Anyone who thinks posting AC makes them secure at the US border is not a valuable poster.

        In fact I call BS on your "most ACs are of a level that their input is valuable". Most ACs I see are posting anonymous snark about MS but don't use MS products. How is that valuable?

        1. Anonymous Coward
          Anonymous Coward

          Re: Title

          I don't snark, I'll post a sarcastic light humoured comment you know like "Your holding it wrong" or something about Office 258, I don't discriminate, I take the piss out of all companies equally when they have a massive fail. I post anonymous because when I'm not trying (emphasis on the "Try") to be funny I also like to post questions or opinions on things and I don't want to be labelled a fool because of my silly posts. As for "karma" points 46.9k up and 7k down so I must be doing something right.

          1. Anonymous Coward
            Anonymous Coward

            Re: Title

            As for "karma" points 46.9k up and 7k down so I must be doing something right.

            Hah! I'm ahead of you on the downvotes. More! More, I say!

            (is this comment useless and snarky enough?)

            :)

        2. Anonymous Coward
          Anonymous Coward

          Re: Title

          In fact I call BS on your "most ACs are of a level that their input is valuable". Most ACs I see are posting anonymous snark about MS but don't use MS products. How is that valuable?

          I'm sort of hoping you also see some valuable anon posts. Not that I would hold myself up as an example due to an exceedingly dark sense of humour, but still :)

        3. Ken 16 Silver badge
          Holmes

          I post anonymous when it bears on my day job

          I find some of the best analysis comes from AC's who are obviously on the inside of the story.

  6. Lee D Silver badge

    Love the "we're losing millions" one.

    Shame, with all those millions, that you didn't think to use another service, another location, a backup, a failover, etc.

    If your restore process is JUST THAT COSTLY, then you need other live hot/warm sites ready to go. All the time. And it'll cost you much less than "millions".

    Whenever I see that, I have so little sympathy for whoever thought that kind of setup was a good idea that I think those people SHOULD be made to pay for their mistakes.

    Azure is one cloud. You could run your own cloud too. You could use another cloud. And you could use multiple datacentres in entirely different continents for each cloud. And then you could join them together and have them failover. It's really NOT that difficult.

    But, nah, just sling Microsoft a couple of grand and obviously everything will be alright even if being down for three hours will cost you "millions".

    1. Bob H

      Absolutely

      Our business has a large part of our capacity in AWS but we have it spread across two availability zones with 100% resilience, two identical and parallel systems designed to balance and scale to the demand. If one goes out then a geographically distinct zone will take over with customers hardly noticing.

      We can't easily expand to different cloud providers because some of the technology we use gets complicated at that level. But as our business expands we are further looking at to what extent we have eggs in baskets and what would happen if the whole of AWS went down (which probably won't happen, normally it will just be a geographical issue).

      1. Ian 7

        Re: Absolutely

        Multiple AZs is a good start, but that wouldn't help you in this case as an entire region went out. AZs help you with issues at an individual data center, but AWS/Azure/GCP roll out changes at a region level so you need to be multi-region to protect you from times when that kind of thing f***s up. And it will. It may be more cost effective for you not to bother, depends on your RTO/RTP requirements.

        When you get into multi-cloud conversations, you're well into the law of diminishing returns.

      2. Claptrap314 Bronze badge

        Re: Absolutely

        Ooooo... You are in TWO AZs? You summer's child.

        You might want to read the book on SRE. Reliability STARTS when you are in three regions, with the DCs on non-overlapping maintenance schedules.

        Anything less, and you WILL have regular outages. If your business can handle those, fine. If not, you better get someone in house that I least knows the basics.

  7. SVV Silver badge

    Actively working to restore services

    Sounds like @AzureBarry is rushing to the data centre right now to put a few more 50p coins in the meter.

  8. Anonymous Coward
    Anonymous Coward

    Magic Rituals

    It added on its status page that "engineers have identified the root cause, and actively working to mitigate the issue."

    I'm guessing mandragore root? In that case, mitigate frog eyes, a hare's leg and magic mushrooms at midnight for best effect.

    1. Rich 11 Silver badge

      Re: Magic Rituals

      We're all used to sacrificing a black cockerel to Lucifer* to get shit going. You know it works.

      *Other gods are available.

      1. hmv

        Re: Magic Rituals

        And I thought it was supposed to be goats; no wonder I never got an answer.

    2. macjules Silver badge

      Re: Magic Rituals

      Engineers addressed the temperature issue, and performed a structured recovery of the affected devices and the affected downstream services.

      Translation:

      "We unfortunately re-arranged the server stacks into a pattern which very closely resembled the dread sigil Odegra. After realising this and and recording temperatures of extreme cold and heat at the same time we made a decision to switch off the power. Unfortunately every website hosted in the North Europe zone will now be displaying 'Hail the Great Beast, destroyer of worlds!'.

      Microsoft apologise for any inconvenience caused"

  9. Anonymous Coward
    Anonymous Coward

    Why is the floor silvery?

    "We've asked Microsoft what exactly an "underlying temperature issue" is."

    Solder's on the floor now. Was in the cabinets before.

    1. whitepines Bronze badge
      Joke

      Re: Why is the floor silvery?

      However, our active Halon based cooling system worked wonders and the glowing air around the cabinet is now at correct operating temperature....

  10. Anonymous Coward
    Anonymous Coward

    Microsoft: 9 myths about moving to the cloud

    https://media.bitpipe.com/io_14x/io_142497/item_1698353/9%20Myths%20About%20Moving%20to%20the%20Cloud.pdf

    Take note of point number 4:

    "Myth 4 Dealing with the cloud is just another hassle"

    "Fact: Office 365 cuts down on hardware and upgrade headaches"

    "When you move to the cloud, the inconveniences of maintaining hardware and upgrading software are a thing of the past. Now your team can focus on being a business rather than a repair service. That means more time spent improving business operations and launching initiatives. Instead of spending ever larger portions of your capital budget on servers for email storage and workloads, you can think strategically and support managers with more agility."

    1. Anonymous Coward
      Anonymous Coward

      Re: Microsoft: 9 myths about moving to the cloud

      Doesn't say you don't have to worry about redundancy, just that software and hardware upgrades will be taken care of. There's nothing in this story that invalidates that statement.

    2. Destroy All Monsters Silver badge

      Re: Microsoft: 9 myths about moving to the cloud

      Instead of spending ever larger portions of your capital budget on servers for email storage and workloads

      Yeah, have you LOOKED at the pricing for MS Server Editions. Luckily there is Linux.

      you can think strategically and support managers with more agility

      It's 2018 and managers can't into strategic thinking. They have to be supported by Morlok brains how can package their ideas with magic agility juice.

    3. macjules Silver badge

      Re: Microsoft: 9 myths about moving to the cloud

      "Fact: Office 365 cuts down hardware and upgrades your headaches"

      FTFY

  11. Scott Marshall

    Underlying temperature issue

    We've asked Microsoft what exactly an "underlying temperature issue" is.

    I'll take a punt that an "underlying temperature issue" is actually the clients getting seriously hot-under-the-collar about the outages.

    1. hplasm Silver badge
      Flame

      Re: Underlying temperature issue

      Underlying temperature issue - see icon.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019