back to article Azure, Office 365 go super-secure: Multi-factor auth borked in Europe, Asia, USA

Happy Monday, everyone! Azure Multi-Factor Authentication is struggling, meaning that some users with the functionality enabled are now super secure. And, er, locked out. Microsoft confirmed that there were problems from 04:39 UTC with a subset of customers in Europe, the Americas, and Asia-Pacific experiencing "difficulties …

  1. Lee D Silver badge

    Another one to add to the bookmarks list I keep for "This is why you really don't want to move off of the on-premises stuff".

    It's getting quite crowded in that folder, to be honest - everything from Azure and Office 365 to Google Apps.

    Sure, use them. But don't rely on them.

    1. fuzzyfelt

      Because on-prem stuff never breaks, floods, gets stolen, overheats or loses power?

      1. Lee D Silver badge

        Didn't say that.

        But when it does, your data is sitting RIGHT THERE. Available to you.

        Literally, pop down to PC World, buy a machine, access your backups, bam... data.

        On-prem is no different to cloud or anything else - but you have your data in your possession. So even if it means "ARGH! Quick... install an exchange server quickly so we don't lose email on our domain", you can do that. Without having to wait for some cloud provider to switch you over. Or you can pull all your existing files and refer to them. Or you can literally make your own mini-network and get things running again.

        While you're tied exclusively into "I have to log in to Azure and it's not working", you can't do a damn thing, even as an admin.

        People with brains do on-prem AND cloud, so failure of one doesn't affect the other.

        I speak as someone who only two months ago had 450KW of three-phase cable arced together at the main transformer station supplying the entire site. Our servers went down hard.

        And I had a copy of the data... just there... literally in arm's reach. Given an absolute emergency, I could have used iSCSI and ANY MACHINE WHATSOEVER to power up some Hyper-V images direct off the storage and booted things up. As it was, it was a weekend and power came back on the Monday, so I just powered up the 50% of kit that wasn't damaged, checked data integrity and carried on as normal.

        But if off-prem doesn't let you even log-in... you're stuffed.

        1. Anonymous Coward
          Anonymous Coward

          > On-prem is no different to cloud or anything else - but you have your data in your possession.

          You don't keep your backups in the *same* cloud, do you?

      2. Sir Loin Of Beef

        No, because when off prem breaks you are relying on someone else, whom you have no control over, to fix your junk in what that person considers a timely manner, to hell with your companies SLA.

      3. bombastic bob Silver badge
        Pint

        The Cloud - highly overrated

        "Because on-prem stuff never breaks, floods, gets stolen, overheats or loses power?"

        Well said! You deserve a beer. And a topic. I hope you like it.

        (I hope you weren't being snarky - which you probably were - but I assume that, on average, the likelihood of any of those things actually happening is pretty low compared to cloudy outages)

    2. Anonymous Coward
      Anonymous Coward

      Moving to AWS

      That's it, had enough, we are off to AWS

  2. Anonymous Coward
    Anonymous Coward

    Damnit!

    Why did we choose to turn this on for the IT department, as a pre-requisite to doing this for all users!

    we should just have started with the guys that were giving their passwords to scammers/spammers first so that if they were locked out it wouldn't be a problem!

    1. macjules Silver badge
      Coat

      Re: Damnit!

      Could be worse, you could have MFA problems AND have your resource_groups located in either Texas or West Europe.

    2. bombastic bob Silver badge
      Meh

      Re: Damnit!

      "the guys that were giving their passwords to scammers/spammers"

      Now everyone's giving those passwords to microsoft, google, amazon, ... "Microsoft Logon" anyone?

      If the cloud were a batch process to backup to it, rather than an ongoing bandwidth-intensive "required to do work" constant access, it would make sense. Cloudy "applications" are not just overrated, they're THE PROBLEM.

    3. FlamingDeath Bronze badge

      Re: Damnit!

      Better yet, take the muppets giving their credentials to scammers outside and unceremoniously shoot them in the back of the head, its the humane thing to do

      Can we do this yet?

      Hopefully soon

  3. Semtex451 Silver badge

    It's worth pointing out that this took out sign in for NHS.NET email admins earlier, as 2FA is mandatory. Their workaround in this instance was to simply to turn 2FA off.

    1. Anonymous Coward
      Anonymous Coward

      >It's worth pointing out that this took out sign in for NHS.NET email admins earlier, as 2FA is mandatory. Their workaround in this instance was to simply to turn 2FA off.

      However, if *all* your admin accounts require MFA authentication it is a little difficult to login to turn it off...

      1. Petey

        Breakglass admin account

        And that's why it's best practice to maintain a break-glass account which is a global administrator. This account does not have 2FA enabled and password maintained in two parts by two custodians.

  4. Alex Trenchard

    I'm glad I'm a dinosaur...

    I've never fully trusted cloud-y things since a previous employer moved everyone to Sharepoint without investing in adequate bandwidth. This was a fairly large premises anyway, but operating at nearly double its initially scoped numbers following multiple rounds of collocations, hot-desking, floorplate adjustments, etc. We had >8,000 people working on a site with an IT backbone scoped for ~4,000 (and that's also 4,000 at the time of build - so far lower planning assumptions than modern services and applications require).

    The fact that I'm old-fashioned and non-collaborative, and regularly make locally-saved working versions of documents means I'm actually able to get on with things today, instead of scratching around and asking "well, what do I do now?!"

    1. h4rm0ny

      Re: I'm glad I'm a dinosaur...

      The fact that I'm old-fashioned and non-collaborative, and regularly make locally-saved working versions of documents means I'm actually able to get on with things today, instead of scratching around and asking "well, what do I do now?!"

      I did some work for a company where the (lone) Sysadmin managed to bork our entire network and consequently almost nobody could work. The director of the company came round and told people they should go home if they couldn't work and that they wouldn't be being paid for the lost time.

      "I have local copies of my files," I promptly said and carried on typing. Others were a little more honest and left.

      The company is no longer there, incidentally.

  5. Anonymous Coward
    Anonymous Coward

    MFAer

    I was planning for a quiet day when my boss decided to turn off and on my MFA.

  6. WhoAmI?

    I'm locked out of my account for work

    so I've cleaned my bathroom instead

    1. DJV Silver badge

      Re: I'm locked out of my account for work

      Ah, but do you paid the same?

      1. Anonymous Coward
        Anonymous Coward

        Re: I'm locked out of my account for work

        "Ah, but do you paid the same?"

        The beauty of being a contractor is that I'm being paid for my time :)

        If I am available to perform the work stated in the contract, and they aren't, well..tough titties.

    2. Peter 26

      Re: I'm locked out of my account for work

      Haha, I cleaned the Kitchen Skylight, been meaning to do that for months..

      Still locked out, what next...

      1. Ken Moorhouse Silver badge

        Re: Haha, I cleaned the Kitchen Skylight

        Simply seeing the Azure sky and the Clouds doesn't help with logging in to it though, unfortunately.

        Compare this with Lee D's second comment above (the case for on-prem):-

        "But when it does, your data is sitting RIGHT THERE. Available to you."

  7. Peter 26

    Ironic

    I was thinking this morning how awful it was that people couldn't get to work because the Waterloo line was shut, and how lucky I am to work from home and not be affected...

  8. Craigie

    SPF

    This reeks of 'single point of failure'.

    What's the point of this cloudy stuff again?

    1. bombastic bob Silver badge
      Trollface

      Re: SPF

      "What's the point of this cloudy stuff again?"

      Fragile house of cards replaces basket for all of your eggs.

    2. hellwig Silver badge

      Re: SPF

      Common misconception. "Cloud" is not synonymous with distributed systems. It makes it easier, sure, but "Cloud" is just the buzz word your accountants use to justify spending less by "co-locating" your data with an "online service provider", "simplifying the support chain", and "reducing non-core business focuses".

  9. royprime

    IT Support said move everything - glad I said no.

    Well, our external IT support suggested we move everything to the cloud, to which I said no, but hey this 2 factor authentication seems like a really good idea as someone brute forced a bunch of our 365 passwords and sent half a ton of spam.

    Now sitting here hitting the refresh as I cannot change any email accounts or login to the portal to set up anyone.

    Great job Microsoft, my F5 key is getting a workout. Still at least I can sit and read El Reg, gives me something to do.

  10. Anonymous Coward
    Anonymous Coward

    Drink Cloudy Bollocks

    An award winning imaginary ale - made of piss and wind and renders everyone completely unproductive.

    1. MyffyW Silver badge

      Re: Drink Cloudy Bollocks

      I've probably drunk worse down the years ...

  11. MatthewSt

    Workaround

    Turns out it's only their MFA validation infrastructure that's flailing. If you whitelist your IP address (which we do for those of us with static IPs) here - https://account.activedirectory.windowsazure.com/UserManagement/MfaSettings.aspx?culture=en-GB&BrandContextID=O365 - then it works fine

  12. SVV Silver badge

    Microsoft have some explaining to do

    Why can changes to Azure that break things not be rolled back almost instantly?

    Why is their testing so poor that these issues cannot be identified?

    Why is customer communication so poor about clearly mission critical systems?

    Why do whole regions get affected when others don't?

    What is your plan of action for improvement?

    Until we hear full, credible answers to basic questions like these, MS cloud services should be considered too unreliable for business use.

    1. Anonymous Coward
      Anonymous Coward

      Re: Microsoft have some explaining to do

      We put MFA on all users with admin rights....I somehow was still logged in from Friday (laptop went into hibernate) and have been lucky to get back in again. Otherwise how would I see this super, helpful message from MS? Thank [insert your deity] we have not rolled this out to all users yet - as per my recommendations to enhance security!?!

      Latest update from Admin Portal:

      MO165510 - Unable to sign in to Microsoft 365 services

      Status:Service degradation

      User impact:Affected users may be unable to sign in using Multi-Factor Authorization (MFA).

      Latest message:Title: Unable to sign in to Microsoft 365 services

      User Impact: Affected users may be unable to sign in using Multi-Factor Authorization (MFA).

      More info: Users may also be unable to carry out self-service password resets.

      Current status: We're in the process of deploying the code update. For those users in the infrastructure that has received the update, we're seeing a drop in error rates and improvement in connectivity. We're monitoring the deployment to ensure the fix is effective.

      Scope of impact: Impact may be experienced by users accessing Office 365 services via Multi-Factor Authentication.

      Start time: Monday, November 19, 2018, at 4:39 AM UTC

      Next update by: Monday, November 19, 2018, at 4:00 PM UTC

      Updated:2018-11-19 14:45 (UTC)

      Start time:2018-11-19 04:39 (UTC)

      1. Ken Moorhouse Silver badge

        Re: For those users in the infrastructure that has received the update...

        Do they mean:-

        For those users in the infrastructure that have received the update...

        Or do they mean:-

        For the user in the infrastructure that has received the update

        1. A.P. Veening

          Re: For those users in the infrastructure that has received the update...

          They mean "the infrastructure, which has received the update", lack of consistent grammar. If it had been for the users, "who" should have been used.

    2. bombastic bob Silver badge
      Meh

      Re: Microsoft have some explaining to do

      Microsoft has NOTHING to explain. They are Microsoft. They are a monopoly, there's noplace else you can go, there's nobody else available, it's ONLY them, you HAVE to accept it, and they don't care about YOU.

      Microsoft don't have to explain NUTTIN to NOBODEEZ. Once you're assimilated, you'll understand...

      [at least, that's the perception]

  13. Anonymous Coward
    Anonymous Coward

    Big Deal...

    Just disable MFA using an admin acco- oh wait we enabled MFA on that one. Oops.

  14. tim 13

    Mitigating

    Not sure that word means what Microsoft think it means

    mitigate

    /ˈmɪtɪɡeɪt/

    verb

    gerund or present participle: mitigating

    make (something bad) less severe, serious, or painful.

    "drainage schemes have helped to mitigate this problem"

    synonyms: alleviate, reduce, diminish, lessen, weaken, lighten, attenuate, take the edge off, allay, ease, assuage, palliate, cushion, damp, deaden, dull, appease, soothe, relieve, help, soften, temper, still, quell, quieten, quiet, tone down, blunt, dilute, moderate, modify, abate, lull, pacify, placate, mollify, sweeten, tranquillize, remit, extenuate, excuse, commute

    "drugs which mitigated the worst symptoms of the disease"

    antonyms: aggravate, increase, intensify

    •lessen the gravity of (an offence or mistake).

    "he would have faced a prison sentence but for mitigating circumstances"

    synonyms: extenuating, exonerative, justificatory, justifying, vindicatory, vindicating, exculpatory, palliative, qualifying, moderating, modifying, tempering, lessening

    "he would have faced a prison sentence but for mitigating circumstances"

  15. Anonymous Coward
    Anonymous Coward

    it ain't just Europe and Asia

    I tried to access email from a certain local, very much on this side of the Atlantic, site, this morning. They had gone all in with Office365, including the imbecilic MFA which requires logging back in every 24 hours and using the MS Authenticator App on your iOS or Android device to re-auth. As they don't pay me enough to care, I had not logged in over the weekend. In the halcyon daze of yore, before they went all MFA, I could just log in, see any email which had accumulated over the weekend, ignore the bumf (which would be 95% of it) and drive on. This morning, of course, their 'secure' email is very secure in most ways. You see, they'd turned the MFA on on desktop/laptop email clients, and on web clients, but not on email clients running on actual iOS devices. Even they weren't so thick as to send MFA alerts to the device the users used to connect to the service requiring MFA. (Hi, Apple; they do exactly that. Exactly what Earthly purpose Apple MFA serves is beyond me.) This means that I can, oh, use MS Outlook on an iOS device and get mail. (One post was from support, telling the users that MFA is borked. Gee. Thanks. We could never have figured that out without you.) Apple Mail is blocked, even on the same iOS devices where MS Outlook works. Hmmm. I wonder why this might be... Web access, including web access to the employee web for everything, is still blocked. Numerous people use the employee web to do actual work; they're locked out as I type this.

    I had posted to my line manager, the vp in charge of this division, and the IT manager back when they first set up this nonsense, saying that this was a disaster waiting to happen. For some reason the IT department is not talking to me this morning. How very sad. I know that certain elements at HR use iPads; I have posted to them indicating that I'll be looking at the next paycheck and I had better not be missing any pay because of this fiasco, copying the little notes I sent to the line manager, the VP, and the IT manager way back when. And, yes, I have been looking for a new job for quite some time, and have found one and will be starting in January.

    Useless gits.

  16. Jay Lenovo Silver badge
    Facepalm

    MF to the A

    Darn, somebody secured the keys to the cloud, in the cloud.

    Anybody got a cloudy coat hanger?

    1. bombastic bob Silver badge
      Devil

      Re: MF to the A

      "Anybody got a cloudy coat hanger?"

      how about a 'cloudy fire axe' ?

  17. Steve Cooper

    My MFA is fine - just checked my logs and have had a dozen or so people since noon get auth'd on our Azure MFA, but I have as much on-prem as possible and only the actual 'contact the end user' bit in Azure.

  18. MAH

    I'm in Canada on a US instance and our MFA is borked....so much for managing any clients instances today.

    Since its unlikely they need to do code changes to fix something that was running fine I suspect they must have changed something and now are trying to fix whatever their eggheads broke.

    1. MAH

      according to the status update they turned it off and on and its now working for me again.

      1. error 13

        Not true, Microsoft don't turn stuff off and back on again.

        They 'cycled impacted servers which has resulted in significant service recovery, with many customers now reporting signs of recovery'

  19. Anonymous Coward
    Anonymous Coward

    With Subtext

    2018-11-19 06:45 (UTC) Scope of impact: Impact is specific to a subset of users who are served through the affected infrastructure.

    "It's just a small number of users"

    2018-11-19 07:56 (UTC) Scope of impact: Impact is specific to a subset of users who are served through the affected infrastructure.

    "No honestly, its just a small number of users, nothing to see here"

    2018-11-19 08:54 (UTC) Scope of impact: Impact is specific to any user who is located in the Europe, Middle East and Africa (EMEA) or Asia Pacific (APAC) regions.

    "It might be a few more users than we thought, but America is fine!"

    2018-11-19 09:59 (UTC) Scope of impact: Impact is specific to any user who is located in the Europe, Middle East and Africa (EMEA) or Asia Pacific (APAC) regions.

    "America is still fine!"

    2018-11-19 12:24 (UTC) Scope of impact: Impact is specific to any user who is located in the North America (NA), Europe, Middle East and Africa (EMEA) or Asia Pacific (APAC) regions.

    "But South America is still okay..."

    2018-11-19 12:50 (UTC) Scope of impact: Impact may be experienced by users accessing Office 365 services via Multi-Factor Authentication.

    "Okay, you got me, it's everyone"

  20. J J Carter Silver badge
    Big Brother

    GDS Lead Security Architect here

    What is this "MFA" of which you speak?

    1. Ken Moorhouse Silver badge

      Re: What is this "MFA" of which you speak?

      I think the "M" stands for "My"

      The "A" stands for "Access".

      Not sure about the "F", any ideas?

      1. eldakka Silver badge

        Re: What is this "MFA" of which you speak?

        I think the "M" stands for "My"

        The "A" stands for "Access".

        Not sure about the "F", any ideas?

        I'll give it a try.

        My Access Fucked

        My Access Fubar'ed

        My Access Floundering

        My Access Fading

        My Access Failed

        My Access Fleeting

  21. Anonymous Coward
    Anonymous Coward

    Workaround for all users

    Whitelist all IP addresses:, since 0.0.0.0 doesn't work use:

    1.0.0.0/1

    128.0.0.0/1

    Which can be done here: https://account.activedirectory.windowsazure.com/usermanagement/mfasettings.aspx

    Then once Microsoft sorts it's act out you can remove the whitelist entries and everyone is back to working normally without having to reconfigure individual users. Beware of course that this increases your risk profile hence why I'm posting this anonymously, I wouldn't advertise that you've done this anywhere

    1. eldakka Silver badge
      Facepalm

      Re: Workaround for all users

      Excellent, let me log in with my admin account to do this, oh, wait.

  22. muhfugen

    US too

    I have some customers in the US who are now locked out of their email accounts on their desktops. Microsoft says its not a big deal because their phone is still working.

  23. Bunker_MonkeyUK

    Can we petition MS to rename it?

    'Cloudy McCloudface'?

    Rolls of the tongue alot better I think?

    1. eldakka Silver badge

      Re: Can we petition MS to rename it?

      Cloudy McBackpfeifengesicht.

  24. Anonymous Coward
    Anonymous Coward

    Latest update

    From MS https://status.office.com

    Current status: Due to the complex nature of this problem, our investigation into the root cause of this issue may take an extended period of time. This incident is and will remain our highest priority until the underlying source of the problem has been identified.

    I’m not looking forward to tomorrow morning...

    1. Sir Runcible Spoon Silver badge

      Re: Latest update

      Normally a change has a 'rollback' option.

      So if they aren't 'rolling back' the change, they obviously have no idea what caused the issue in the first place. Unauthorised change/hack?

  25. Anonymous Coward
    Anonymous Coward

    I'd just finished watching the ignite session

    My thoughts were..looks interesting. Nowhere near ready for prime time!!

    https://www.youtube.com/watch?v=7hoEmEwV8Rk&feature=youtu.be

    1. Dan 55 Silver badge

      Re: I'd just finished watching the ignite session

      Nowhere near ready for primetime? Bet it's already bring deployed to production then!

  26. LateAgain

    Naturally there is a refund...

    A refund for time that the services paid for are unavailable is, of course, going to happen.

    1. ricardian

      Re: Naturally there is a refund...

      Latest:

      11/19

      Issues connecting to Azure resources in Europe, Asia and the Americas regions using Multi-Factor Authentication

      Summary of impact: Between 04:39 UTC and approximately 21:30 on 19 Nov 2018, customers in Europe, Asia-Pacific, and the American regions may have experienced difficulties signing into Azure resources, such as Azure Active Directory, when Multi-Factor Authentication (MFA) is required by policy.

      Preliminary root cause: Requests from MFA servers to Redis Cache in Europe reached operational threshold causing latency and timeouts. After attempting to fail over traffic to North America this caused a secondary issue where servers became unhealthy and traffic was throttled to handle increased demand.

      Mitigation: Engineers deployed a hotfix which eliminated the connection between Azure Identity Multi-Factor Authentication Service and a backend service. Secondly engineers cycled impacted servers which allowed authentication requests to succeed.

      Next steps: Engineers will continue to investigate to establish the full root cause and prevent future occurrences. A full Root Cause Analysis will be published in approximately 72 hours. To stay informed on any issues, maintenance events, or advisories, https://www.aka.ms/ash-alerts and you will be notified via your preferred communication channel(s): email, SMS, webhook, etc.

  27. tmc_itdude
    WTF?

    Admin Mobile App

    Like many we were caught out with this MFA issue and find it difficult to understand the poor communication and duration it took to fix. Found a way around it using the Office365 admin Mobile app, sorry late in the day to be posting a work-around.

    The app still functioned and remained logged in unaffected by the MFA issue for user admin/email tasks including creating new users. We created a new user and assigned Admin role, it is not possible to enable or disable MFA in the app. This is fine as once user created we could log in with the new Admin and turn off MFA for the existing Admins.

    Although a bit slow, the Admin app is quite handy for lots of things and got us out of a hole yesterday.

  28. Wayland Bronze badge

    Office 360 strikes again

    They will have to call it Office 359 next year.

    1. Dan 55 Silver badge

      Re: Office 360 strikes again

      Don't you mean next week?

  29. streaky Silver badge

    Office..

    Outlook (email) was also down completely for 4 hours the other night and nobody noticed. When we were acquired by our new parent company we were forced off the perfectly fine corp google mail service we were using that never had issues onto the Microsoft stuff and it's been crap ever since. I usually defend Microsoft quite a lot but you'd think they could manage to keep something so simple online for everybody, all the time.

    1. Sir Runcible Spoon Silver badge
      Facepalm

      Re: Office..

      I usually defend Microsoft quite a lot

      The mind simply boggles -->

      1. streaky Silver badge

        Re: Office..

        Because the criticisms are normally unfair.

  30. Anonymous Coward
    Anonymous Coward

    hey MFA trials with Azure cloud, testpilot, IT do you want to trial?

    now they know why i said id rather shave my giblets with a broken beer bottle, itll end in tears you tits!

    cue 90% of global IT up the shitter today, heh, didnt think it would only take 10 days to prove my point.

    TODAY was ace fun at work :P on-prem ftw! HA failover setup MS? Offered C&C use of some of my rackpsace servers twiddling their thumbs, oh your MFA`d ALL the ADM accs inc the master one, a good job done well my stateside brethren! <cue mic muting and pissing IT pants UKside>

  31. khjohansen
    Devil

    ..by their own petard??

    AzureSupport not responding 'cause they can't access their accounts?

  32. FlamingDeath Bronze badge

    I really cant understand

    Why Microturd are market leaders

    Oh now I remember, they gave their shitware away for free in the early days to get market share and now we’re all fucking stuck with them, all businesses use their clusterfuck software, its a fucking shitshow

    Someone need to tell them to fuck off and chew on a tailpipe

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019