back to article Good guy, Microsoft: Multi-factor auth outage gives cloudy Office, Azure users a surprise three-day weekend

Microsoft is battling to fix its knackered multi-factor authentication system that today blocked customers from logging into their Microsoft 365 and Azure services. The Redmond giant confirmed on Friday an unspecified glitch prevented customers in North America from receiving the multi-factor auth (MFA) codes they need to sign …

  1. Claptrap314 Silver badge

    Office 360

    FIFY.

    It's good all-around software, dontcha know.

  2. Crazy Operations Guy Silver badge

    11 months to the day...

    I figured they would've learned from when this exact same thing happened exactly 11 months ago.

    So glad we tore up our contract with Microsoft last year when this happened and moved to an on-premise install of Dovecot with some calendaring plug-ins and switched from Outlook to Thunderbird with a Calendering add-on. We also noticed that since moving to a cloud-based solution, our workers became increasingly stressed and work output dropped due to workers responding to emails and messages at all hours. Shortly afterwards, we implemented a "Work is for work hours, all other time is for personal things" policy and work quality has increased sharply. People are reporting to work well-rested and stress-free. We have people to handle the off-hours stuff (6 shifts; 3 full-time weekday shifts and 3 part-time weekend shifts). Management is happy since we're spending far less on customer support and increased sales from customers happy with our now-improved products.

    1. Anonymous Coward
      Anonymous Coward

      Re: 11 months to the day...

      10,000 staff and using Office 365 happily, not sure how few users you have but to move 10,000 you'd still have it cloud or DC hosted.

      1. Crazy Operations Guy Silver badge

        Re: 11 months to the day...

        A little over 300 users. I suppose you're happy with it because your company forks over enough money to get Microsoft to actually pay attention to your problems. For us, it feels like Microsoft doesn't give a shit since its cheaper for them to compensate us for violating the SLAs rather than paying staff overtime to comply with the SLAs. Sure, that means we are essentially hosted for free, but with all the problems we've had, free is too expensive.

        We have our own datacenter and space is essentially free. It was built to house our System/360 and the other computers we'd buy in the coming years. So now we have a 3600 sqft room for the 250-ish active systems we have. We also have a very power intensive factory attached to our building, so we get our electricity at a bulk discount

    2. cb7

      Re: 11 months to the day...

      The only problem is Thunderbird isn't quite industrial strength. It slows down considerably once you get to multi GB mail storage (a necessity for some businesses) vs MS-Outlook on the same hardware. Searching, sorting and general navigation all get much slower on TB.

      1. Ken Moorhouse Silver badge

        Re: It slows down considerably once you get to multi GB mail storage

        There's a trade-off though, speed vs corruption. The bigger the file store, the more it is susceptible to corruption. In my experience TB is a lot less susceptible to corruption than Outlook. Put another way, Outlook pays less heed to boundary conditions than TB. Microsoft appear to acknowledge this...

        "Note that .pst files are not meant to be a long-term, continuous-use method of storing messages in an enterprise environment." (Microsoft).

        The crazy thing about email is that it really should be stored in a proper SQL database for the best of all worlds, speed, integrity, security and accessibility even over LAN/WAN links. Archive utilities do of course exist but the ones I've seen are not intended to replace the email client.

        1. The Original Steve

          Re: It slows down considerably once you get to multi GB mail storage

          Not disagreeing with your headline point about TB being less susceptible to corruption than Outlook.

          But...

          1. PST files are a PERSONAL export of mailbox content. Outlook uses an OST file as an offline cache, but it doesn't use PST files for anything unless you export content from your mailbox to a PST you create. PST's are really only a thing for end users to do their own, manual archiving and is not recommended in enterprise scenarios.

          2. Exchange uses a relational DB for its mailbox store. A cut of what was the JET database. Last 4 or so releases of Exchange have been very reliable in terms of mail store.

          3. As PST's aren't used, corruption claims don't apply. Should your OST become corrupt, simply delete it. Only your offline cache, it'll get automatically rebuilt.

          4. Both Exchange on premise and Exchange Online have an archive feature, which essentially adds another mailbox for each user for archive purposes. Users can simply drag and drop content from primary to archive, or admins can create rules.

          YMMV, and I wish you luck with TB, but personally I'd take Outlook as a heavy duty mail client in an enterprise over TV any day of the week.

      2. Crazy Operations Guy Silver badge

        Re: 11 months to the day...

        Most of our staff don't keep that much mail in their boxes, we discourage massive signature, and attachments are done by linking to the file instead of including it in the message itself. Plain text messages are also fairly common (Some people here joined the company before HTML mail was a thing, a few of the more senior staff learned to use computers during the bang-path days)

        We've also offset the performance hit by removing the overhead that Windows was putting on the systems. Aside from the engineering staff (Who make up 60% of the company and were already running Linux), everyone's just been using browser-based applications hosted on local servers for their work, so after a little retraining, very few people have complained about the switch. At the very least, the switch was less painful than when we switched to Windows from Solaris back in the late 90s.

        We don't really need much in the way of newer technologies, we manufacture, and support, machines for other manufacturers. We started building machinery for these new-fangled "Assembly Lines" that are all the rage and now make complex machinery that our customers use to build their products. So like our most popular product is a set of machines that cut, fold, and weld metal sheets into metallic cases for other products. We like to sum up our company as being a "Factory factory".

    3. PrudentIsMyBlood

      Re: 11 months to the day...

      It looks like that some people like only destructive and biased ranting more than constructive criticism...

      You did say whatever you wanted to say but the fact is that Microsoft is loved by billions not because they pay people to like them but because Microsoft is best in almost everything they do.

  3. Mephistro Silver badge

    I'll add this article...

    ... to my ever growing list of reasons for NOT using Orifice 365. Very useful when one of my clients asks for my opinion on the possibility of acquiring and using said product.

    Although the part about the extended weekend sounds nice...

    ;^)

    1. Ken Moorhouse Silver badge

      Re: I'll add this article...

      I've lost customers due to my "old-fashioned" views on technology.

      The irony is that I have spent (and continue to spend) the majority of my working life on R&D type projects. (Grant-aided where appropriate).

      My philosophy continues to be that mission-critical customer data, and its fail-over means of access and control needs to be achieved via on-prem means.

  4. coconuthead

    authenticator app does not "receive" codes

    TOTP authentication does not work by the site "sending" a code anywhere. The code is synthesised on the device (user's phone) from the current time and an initial value set when the authentication was set up. I don't know whether Microsoft's authenticator app also offers some other mechanism, but I use Google's Authenticator app for my Micorosft 2FA, so they do support standard TOTP.

    The linked Microsoft announcement says "Users may not receive authentication requests via phone call, SMS or within their authenticator app." Perhaps they meant something internal to their infrastructure was not receiving requests. Or perhaps they meant to write "replies", as The Register assumed, and it is wrong about TOTP access being disrupted.

    1. solv

      Re: authenticator app does not "receive" codes

      The microsoft authenticator app receives a push notification from Microsoft asking if you are trying to login to which you click yes.

      Google does this as well if you ask it to, however it also lets you do standard TOTP as a fall back in case push doesn't work.

      So if their services were down then they couldn't initiate the push to your authenticator app.

      1. Dan 55 Silver badge

        Re: authenticator app does not "receive" codes

        There is, however, an option to use a non-MS authenticator app and there might be a slim chance that that is still working.

        1. Pascal Monett Silver badge

          Great ! Just log in and configure that in your profile settings.

          Oh, wait . .

    2. Ken Moorhouse Silver badge

      Re: TOTP

      There's the problem... they should be using IMAP instead...

    3. RuffianXion

      Re: authenticator app does not "receive" codes

      TOTP? Wasn't Top of the Pops cancelled years ago?

    4. Anonymous Coward
      Anonymous Coward

      Re: authenticator app does not "receive" codes

      Pick nits much?

      I'll tell you that during the outage, i was unable to use the generated code from my handset.

      Nor was I able to receive the MS Authenticator app requests to approve sign in from my handset either.

      The app does indeed receive requests for approval from the users' handsets and will use that instead of a user keying in a numeric key.

      as far as any user would be concernet, it's a request/response mechanism. becaues, you know, it does receive requests.

      Since, as you admitted, you don't know how the app works why don't you make like Bambi's Flower, or at least speak not of which you know not.

  5. WolfFan Silver badge

    It’s still slow

    I had to re-up my 2FA late last night; it took literally five minutes to go through. As of a hour ago it took at least four minutes to go through for someone else who had to connect to the same system. This would normally have resulted in timing out; that it didn’t indicates that there’s still a problem and they know it.

    1. Mark192

      Re: It’s still slow

      I wonder if they've got a team frantically doing things manually...

  6. Drew Scriver Bronze badge

    El Reg Fail - being locked out is NOT security

    The sub-headline states "Redmond's sign-on system is so secure, nobody can get in".

    Some will say I'm picking nits, but I'm tired of hearing execs say that draconian measures equal security. They don't need help from El Reg to bolster their lack of understanding.

    I'm of the old-school notion that security requires three components: confidentiality, integrity, and availability. Since the latter was violated in the MS outage security suffered.

    Be right back after rebooting my laptop because of yet another middle-of-the-day forced update that some exec believes is essential to provide 'iron-clad security'.

  7. Anonymous Coward
    Anonymous Coward

    Got to ask your business: what if the outage was longer! how would the business recoup lost revenue/productivity.

    Last time I looked: Office / e-mail (comms) was considered a critical service. If you are medium to large size business why would you place a critical system in someone elses hands?

    But its ok, All these cloud providers have all the certifications against all the standards !

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019