back to article Microsoft Office 365 and Azure Active Directory go TITSUP*

It seems Microsoft's Office 365 is having an unscheduled nap as users across the world report difficulties logging into the administration portal. Office 365 is having issues since 04:51 AM ESThttps://t.co/3GGuh8EZs8 RT if you're also affected #office365down pic.twitter.com/ixHgD7YHv1 — Outage Report (@ReportOutage) April 6, …

  1. Sil

    Email and OneDrive

    I couldn't access OneDrive and Exchange for about 1-2 hours today. Everything's back to normal.

  2. J. R. Hartley Silver badge

    Re: Email and OneDrive

    "Microsoft in SNAFU Shocker"

  3. Christian Berger Silver badge

    Why???

    I mean apparently people use this for e-mail and file servers. Both aren't particularly hard things to do. How can someone mess that up so badly?

    Or to rephrase the question: How can we learn from this?

  4. defiler Silver badge

    Re: Why???

    Because it tends to be cheaper and easier than running your own resilient email system, if you're only managing a handful of users. Sure, it goes off from time to time, but that passes and in a few days it's forgotten about.

    It's like when you drink tea that's too hot and it burns a bit on the way down, but a few seconds later you're fine and keep drinking the tea once it's cooled a little. People are funny like that.

    Substitute coffee / hot chocolate / hot beer / hot absinthe or whatever you like to drink hot in lieu of the tea...

    We run our email in-house, over multiple servers, in multiple sites.

  5. Lee D Silver badge

    Re: Why???

    Don't use it as your ONLY resource, I think should be obvious.

    Such cloud options are great. As one option per cloud. But you should have in-house stuff too, or you're entirely reliant on a) your Internet connection and b) Microsoft. There's no reason you can't run a secondary email server, AD server, etc. in-house. If you do it right, I can't see a reason that external users would notice anything had even happened if the entire cloud went down.

    But it's "lazy IT". Let's just pay a monthly subscription, then that's us sorted because "it's Microsoft". They don't give any thought to business continuity. Sure, it can work at some levels of business. But if you are a corporation of any size and you're affected by this,.. you obviously put too many eggs in the same basket.

  6. Anonymous Coward
    Anonymous Coward

    Re: How can we learn from this?

    Many other people had already learned about this, long before the latest occurrence. (Based on what I've seen from you here, you're one of the people who saw this coming).

    So perhaps the question that really needs answering might be "why does persistent IT incompetence on this scale (and with no improvement in sight) seem not to matter to the people who pay the IT budgets?"

    NB "IT budgets" also needs to include anything where computers and software play an important part of product functionality, e.g. automation equipment, 'self driving cars', a whole pile of computer-dependent stuff.

    Suggestions welcome.

  7. Anonymous Coward
    Anonymous Coward

    Re: How can we learn from this?

    Question:

    "why does persistent IT incompetence on this scale (and with no improvement in sight) seem not to matter to the people who pay the IT budgets?""

    Answer:

    Most high level management these days is basically a bunch of gamblers who are taking the gamble that the reductions in resilience (or in MBA speak.. "savings") that generate large bonuses for them will not come back to bite them in the arse before they retire or move on to the next schmuck of a company that believes their interview bull.

    Simples

  8. Christian Berger Silver badge

    Re: How can we learn from this?

    "why does persistent IT incompetence on this scale (and with no improvement in sight) seem not to matter to the people who pay the IT budgets?"

    It is, perhaps, because people who know how computers work, generally don't work in computer administration any more, they move on to higher paying jobs like programmers.

    There are few exceptions like that game company where the employee handbook leaked recently. They had a system where you had desks with wheels you could move around by yourself. As soon as you plugged it in on your new location the floor plans would automatically get updated.

    Since most IT-departments are horribly bad at what they are doing, most people have never experienced a good IT-department, which means that they don't demand it to be actually usefull. Good IT is to rarely seen as an enabler of success and an effective motivator of your staff.

  9. Christian Berger Silver badge

    Re: How can we learn from this?

    "...that generate large bonuses for them will not come back to bite them in the arse before they retire or move on to the next schmuck of a company that believes their interview bull."

    I don't think it's that. I mean most bad IT decisions don't actually save any money, not even in the short run. In fact many even have short run negative consequences.

    My hypothesis is that there are many IT departments which believe _anything_ a salesperson will tell them. That's why companies _still_ run antivirus software, even though it's benefits have long been disproven by both theory and practice. That's why companies still invest in "office productivity" software like Microsoft Office or Open Office, even though those mostly cause your employees to waste time on things they don't know how to do, like making a printed document look good.

  10. CrazyOldCatMan Silver badge

    Re: Why???

    hot absinthe

    Does the wormwood degrade if you heat it? That's the only reason why people drink the Green Fairy..

  11. hplasm Silver badge
    Holmes

    Re: Why???

    An abomination.

    Never light the sugar.

  12. Anonymous Coward
    Anonymous Coward

    Re: How can we learn from this?

    Because the people that are paying the IT budgets like shiny things and savings. They are fooled by so called digital directors who are just hipsters in disguise with no clue about real IT work. These so called directors want to be "Infrastructure free" because it will be a "massive cost saving", clearly showing they haven't a clue about how the cloud or local IT actually works.

    Problem again is, the people that pay the budgets listen to this fool.

  13. admiraljkb
    IT Angle

    Re: Why???

    @Lee D

    "But it's "lazy IT". Let's just pay a monthly subscription, then that's us sorted because "it's Microsoft". "

    That 2nd sentence is an MBA laden CxO talking, not rank and file "proper" IT. IT much of the time doesn't make its own budgets or decisions on stuff like this. A couple of dinners with some MS sales guys at the CxO levels, and this sort of thing gets pushed through over IT's objections. (been there, got the t-shirt). I would say it's probably more Lazy management that's trying to trim both the personnel and equipment budgets while simultaneously getting nice meals...

  14. Milton Silver badge

    Re: How can we learn from this?

    'So perhaps the question that really needs answering might be "why does persistent IT incompetence on this scale (and with no improvement in sight) seem not to matter to the people who pay the IT budgets?"'

    Because they collected their handsome bonus for "cost saving measures" and left the company before the consequences became obvious. They're the same people who were happy to be sold a bunch of outsourcing crap; the same ones who paid small fortunes for endless streams of suits wrapped around MBAs from three- and four-letter professional-grade-bullshit (PGB) outfits, to be told, during an elaborate Death By Powerpoint, how to synergistically leverage the business process enhancement matrix for maximal mission-driven shareholder return which, strangely enough, meant cutting costs by sacking the very few remaining greybeards who knew how things actually worked.

    In short, "the people who pay the IT budgets" have long since learned to trouser the bonus and skip out before the scale of their monumental ignorance and disastrous incompetence become apparent. Many of them have CVs to leave you leave you breathless with admiration, when you know that they have had 20-year careers fucking up every single thing they have touched.

    But do not imagine that they are entirely worthless meat: to the saleslizards who work for the aforementioned PGB consultancies, such executives are as a herd of plump gazelle to the gaze of drooling hyenas ... fat, dumb and oh so tasty.

  15. Fruit and Nutcase Silver badge
    IT Angle

    Re: How can we learn from this?

    @Christian Berger

    It is, perhaps, because people who know how computers work, generally don't work in computer administration any more, they move on to higher paying jobs like programmers.

    I have come across programmers who appear not to know how computers work.

  16. Arctic fox
    Childcatcher

    @ Christian Berger RE: "Since most IT-departments are horribly bad at what they are doing"

    You post that here at El-Reg? You enjoy getting hosed with downvotes? Your masochistic inclinations should be a private issue. See icon. :P

  17. Anonymous Coward
    Anonymous Coward

    Re: @ Christian Berger RE: "Since most IT-departments are horribly bad at what they are doing"

    getting hosed with downvotes

    It perfectly demonstrated the types of people who frequent this forum.

  18. Joe Montana

    Re: Why???

    The keyword is "resilient", but the frequency of outage reports posted here seems to suggest that it isn't really very resilient at all...

    Running an unreliable isn't very expensive or difficult.

  19. Joe Montana

    Re: How can we learn from this?

    The problem is that the people making the purchasing decisions don't understand technology at all, so they don't question what sales people or random websites are telling them.

    Most such purchasing decisions are not made by the IT department, but even the IT dept often don't have much of a clue either. The requirement for staff increased much more quickly than the availability of skilled staff, so companies have to take whatever they can get - including people who don't have much of a clue.

  20. Dr Dan Holdsworth Silver badge

    Re: How can we learn from this?

    Unless you can actually check each item on a CV for veracity, it is wise to assume it to be at best wildly inaccurate and a heavily manicured version of the truth, and at worst complete fiction. By way of example, a former colleague of mine describes the time when we worked together (as low-end dev-ops on a Remedy ARS system) as the time when he was the team leader of that entire section.

    Which, to be brutally honest, was complete and utter balderdash, wild exaggeration and outright fabrication of what he was actually doing.

    Employing a person on the basis of information that they provide which cannot be independently checked is utter folly.

  21. Michael Wojcik Silver badge

    Re: @ Christian Berger RE: "Since most IT-departments are horribly bad at what they are doing"

    It perfectly demonstrated the types of people who frequent this forum.

    Usage pedants, right? Because Christian used "it's" where "its" was wanted. That's the only reason I ever downvote anyone.

  22. rmason Silver badge

    All brilliant Ideas in theory (not relying on the cloud, or any single source).

    I suspect my situation is like many others.

    Can you imagine what the answer would be (or in fact, was) when you ask for all the relevant kit, software and licences to have an on-prem exchange box (for example) *AND* the cloud offering, when the previous iteration of people in your job sold them on "the cloud" to save money?

    I'll give you a hint. Two letters, started with "n", ended with "o".

    The upshot was we've had approx 1,5 hours of slightly flaky emails, and skype refused new log-ons for a bit (but worked for those logged in). Now all fixed (10ish-12ish today).

    In our business neither thing being down was a significant cost or risk, so correct decision made (as those who said "no" would see it, they're probably right).

  23. Lee D Silver badge

    Oh, I'm quite sure there are tens of thousands of places that do just that.

    Cost-benefit analysis is fine, so long as someone did it.

    It's when people whine that "Oh, my 4G credit card reader is down, I can't take cards, my business is in ruins" or "WE CAN'T GET OUR EMAILS!", when they haven't bothered to take such a loss into account that bothers me.

    To be honest, everything in the world from GMail to Azure, AWS to IBM will be down for things on that order of magnitude, no matter what they promise. They have to be. But it's what YOU do about it for your circumstances that matters.

    Emails will delay an hour or so and then come in later if it comes up. Skype shouldn't be a business-critical tool. Your remote workers not being able to get in on VPN for a bit is no worse than someone tripping over a plug. But when you whine that your work-at-home telesales can't dial into your VoIP VPN for a fraction of a second and it's costing you money, I have to just think "Okay, so what was your backup?"

  24. rmason Silver badge

    @Lee D Absolutely.

    In my Yoof I worked for, cough, *a* yorkshire based ISP.

    With every single outage would come the angry phone calls and even speculative invoices from people who "can't operate my business" and are "losing thousands an hour here".....

    All from their 9.99 a month (single) home ADSL account that happened to be down.

    *shrugs*

    Same deal innit.

  25. admiraljkb

    @rmason

    o365 isn't actually cloud. It's hosted. *IF* if was actually cloud, it would be more resilient and less susceptible to the issues that keep downing it. :)

    Sidenote - In a meeting with MS o365 and Azure Sales and Tech, the o365 drone said "cloud", and I corrected him, with the Azure guys nodding with me. :) They weren't happy about the o365 guys throwing the word "cloud" about when they are a hosted service...

  26. Amos1

    There is no "cloud". It's just someone else's computer, as Orifice 365 definitely is.

  27. admiraljkb

    @Amos1

    Hosted is someone else's computer - and o365 is totally hosted, with little in the way of resiliency.

    Strictly speaking, Cloud is quite dynamic/automated and much more resilient and distributed. However... Marketing weenies will call everything on the internet "cloud" to keep a marketing hypewagon of buzzwords rolling. So at this point, the term cloud is virtually meaningless...

  28. Wayland Bronze badge

    On the Plus side

    maybe they will upgrade to business service?

  29. Dr Dan Holdsworth Silver badge

    Buzzword Bingo

    Cloud is a buzzword, simple as that. No real meaning, except what that shiny-suited salesman wants it to mean.

    A cloud-hosted solution is a data centre full of servers on which the service is running. The only difference is that the cloud-hosting operator can probably do it a bit cheaper than you can, through economies of scale. The *other* difference is that you are not only relying on the cloud hosting machines to stay up, but also all the networking kit between you and them, plus other associated gubbins like DNS and the like.

    When it all works, it is cheaper. When it doesn't, what ho, you got what you paid for.

  30. nsk1974

    *ALL* Microsoft hosted services were inaccessible for our business during this outage. Skype for Business, OneDrive, SharePoint Online, Outlook etc. Much more disruptive than just the publicised "Can't access the admin portal for Office 365" statement. Communication from Microsoft during this major EMEA wide outage has been very poor.

  31. hplasm Silver badge
    Devil

    Blue Sky Thinking.

    Depending on The Cloud is bad enough.... but Azure?

    That's the blue bit in between Clouds - there's nothing there!

  32. steelpillow Silver badge
    Trollface

    What a great advert for ...

    ... Chrome OS? except, limited choice of desktop workstations

    ... MacOS? except, stupidly expensive

    ... Linux? except, blinded by choice and FUD

  33. big_D Silver badge

    Re: What a great advert for ...

    ChromeOS wouldn't have been any better, had its background services gone down...

  34. hplasm Silver badge
    Windows

    Re: What a great advert for ...

    "... Chrome OS? except, limited choice of desktop workstations

    ... MacOS? except, stupidly expensive

    ... Linux? except, blinded by choice and FUD"

    Microsoft - All of the above, and worse?

  35. hplasm Silver badge
    Gimp

    Re: What a great advert for ...

    "ChromeOS wouldn't have been any better, had its background services gone down..."

    But, it didn't... unlike the usual suspect.

  36. 404 Silver badge

    Re: What a great advert for ...

    If you could compile a linux kernel that ran Windows x86 programs natively, you'd be Bill Gates rich...

    Ah, if wishes were horses... I wouldn't be sweating the upcoming Windows 10 Apocalypse Edition when support for Windows 7 goes away.

    :\

  37. Anonymous Coward
    Anonymous Coward

    Re: What a great advert for ...

    I don't see anyone throwing money at the wine project.

  38. Wayland Bronze badge

    Re: What a great advert for ...

    WINE, Crossover Office are experts in this area.

  39. big_D Silver badge

    BOFH

    It would appear that Redmond has opted to secure user data by, er, removing access to it entirely. Clever.

    Fast a BOFH thing to do on Friday... Let me guess, the phones are also off the hook in operations?

    Although if it was really BOFH, he would have just deleted the accounts...

  40. pdh

    Sometimes it's the right choice

    > as those who said "no" would see it, they're probably right

    Yes. If you're big enough to afford a proper IT staff, and the necessary software licenses, and redundant hardware, then maybe you can do it better in-house. But for smaller outfits the cloud often makes more sense.

    Sure there's going to be occasional downtime, no matter how many 9's your cloud provider claims, and of course you still need to keep backups somewhere other than in that same cloud, but unless you can afford multiple competent IT people (more than one, since they tend to take vacations sometimes) and redundant hardware (elsewise you have a single point of failure in your one crucial server, and how long will it take to re-build the server when it eventually fails) you can probably get better uptime in the long run if you use cloud solutions.

    It's like mains power: a few sites really do need a large onsite generator and etc so they can ride out multi-hour power outages, but that's very expensive. So most people just have enough UPS so they can run for a few minutes and then safely shut things down if the power doesn't come back quickly. Long power outages are rare enough that for most businesses it doesn't make economic sense to maintain your own fully-capable power-generation equipment. So it is becoming with IT.

  41. onceuponatime

    Re: Sometimes it's the right choice

    What is this "vacation" of which you speak? You keep using that word, I do not think it means what you think it means.

    For a small business Office 365 is more cost effective than running exchange. If you get more than about 25-30 users it becomes less so over time. Problem is convincing people to spend the dosh to do internal servers in a CAS array for a small business is not going to happen most of the time.

  42. a_yank_lurker Silver badge

    Re: Sometimes it's the right choice

    The cloud is not always suitable for small businesses either. One has to consider data privacy laws such as the US HIPPA that have strict requirements about data access, etc. The cloud is also nothing more than moving an internal server farm to an external server farm with supposed cost savings (or more accurately more obvious costs) and possibly some operational efficiencies.

    One major difference with external vs internal is how many are affected when the service goes down. Internal only affects one company while external affects numerous.

  43. Fruit and Nutcase Silver badge

    Re: Sometimes it's the right choice

    @pdh

    Yes. If you're big enough to afford a proper IT staff, and the necessary software licenses, and redundant hardware, then maybe you can do it better in-house.>

    I worked for a client who are not a small outfit by any means. They went over to o365, and I think was the right decision, even though at the time I did not think so. And that opinion is purely from having subsequently seen the inertia that appears to be endemic in their IT department. So, if Microsoft can deliver a better service, good for them - the in-house IT only shot themselves in the foot, IMHO

  44. TomG

    Re: Sometimes it's the right choice

    Finally someone used the phrase "server farm". I thought this designation was gone forever.

  45. ruscook

    My first thought too - the new security is working :-)

  46. Blockchain commentard

    Didn't notice any problems at all this morning. Possibly because I'd told Microsoft to locate our stuff on UK servers due to impending GDPR requirements.

  47. Doctor Syntax Silver badge

    "Possibly because I'd told Microsoft to locate our stuff on UK servers due to impending GDPR requirements."

    Not that that's going to do you much good. As far as the US govt and the CLOUD act are concerned there's no such thing as UK servers, just servers.

  48. Anonymous Coward
    Anonymous Coward

    On the main tenant we manage we never experienced any issues. Maybe because we run a hybrid setup and adfs authentication, we’ll see when (or if) they post an RCA.

    Another tenant gave an error that said something along the lines of “could not authenticate at the time”, but a third one (and the first one I checked when calls came in 2 to 3 hours ago) said “user unknown on this system”. I very nearly shat myself.

    It seems the crisis has passed, and fortunately before beer o’clock. Another notch on the “bad microsoft” stick, but I still wouldn’t go back to running it all in house... the absolutely critical, losing-money-if-its-down stuff : yes. Email ? IM ? Sharepoint ? Hell no. I wouldn’t be able to get the uptime of most cloud providers without spending huge amounts of dosh.

  49. wolfetone Silver badge
    Headmaster

    Remember Kids

    The "cloud" is just someone else's computer.

  50. Jacques Kruger

    Well, in 28 months this was our first perceivable outage. Not bad, and a hell of a lot better than any other email service we've used in the past.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2018