back to article Microsoft struggles against self-inflicted Office 365 IMAP outage

Microsoft engineers are struggling to fix a seven-day-old, self-inflicted Office 365 IMAP outage. IMAP access to Office 365 tanked on January 18, meaning customers could not access emails using Exchange Online via IMAP or connect third-party mail clients via IMAP. Microsoft told disgruntled Office 365 customers that the …

Page:

  1. Johan Bastiaansen

    who could have thunked?

    Ha ha ha ha ha ha ha ha ... [wipes away tears] ... ha ha ha ha ha ha ha ha...!

    1. Wzrd1

      Re: who could have thunked?

      NT4 SP6...

      I guess they're still not testing their products correctly.

  2. Erik4872

    Is part of the problem all that abstraction?

    Every new project I've been involved in is many, many layers divorced from the real software, OS and hardware that it runs on. Is it possible that there's so much high-level stuff involved in Azure that directly managing each O365 customer as a discrete Exchange instance isn't really possible anymore? They mention a failed update, but you would think an update to Exchange could be rolled back. Unless that update had a knock-on effect to other software-defined stuff for that customer, that is...

    1. Howard Hanek Bronze badge

      Re: Is part of the problem all that abstraction?

      Abstraction.....obstruction.....

      What difference does it make!

      To quote a well known figure......

  3. Howard Hanek Bronze badge
    Joke

    Untapped Market

    The Three Stooges are really ripe for a comeback don't you think?

    Let's modernize them and depict them working at a technology company, in Redmond, in product development......

    -or-

    Technical Support troubleshooting the Cloud......

    1. chivo243 Silver badge

      Re: Untapped Market

      It's sabatoogie I tell ya! Whoooo whoooo whoooo whoooo

    2. Pedigree-Pete Bronze badge
      Angel

      Re: Untapped Market

      Wot. Like "The IT Crowd"?

  4. Anonymous Coward
    Anonymous Coward

    Voicemail

    What about the 16 day outage we had on office 365 voicemail? yeah that was fun.

    1. Wzrd1

      Re: Voicemail

      We're slowly migrating to O365 from Bloatus Notes.

      Looks like there'll be a bit of a delay.

      Whatever was wrong with sendmail?

  5. Anonymous Coward
    Anonymous Coward

    So are they going to rename it Office 358?

    1. PJF

      So are they going to rename it Office 358

      More likely to 3.653....

      as to up-time per 24 hrs...

  6. Voland's right hand Silver badge

    Not like it worked even before that

    IMAP support in both MS Exchange and Office 365 is a complete and utter joke.

    All people who actually need to have it working use davmail. That works (TM).

    1. Destroy All Monsters Silver badge

      Re: Not like it worked even before that

      Because IMAP is extremely complex!!

      No, wait...

    2. Wzrd1

      Re: Not like it worked even before that

      I've found that sendmail works quite well. Even after patching, it still works.

  7. Lusty Silver badge

    No rollback? I thought that was the first principal of continuous deployment?!

    1. petur

      What did you expect? This the 'Cloud', you pay money, have zero control and maybe get some service...

    2. Anonymous Coward
      Anonymous Coward

      @Lusty - Nah, this is DevOps stuff

      They can't be arsed to roll back so they keep rolling over hoping they might get it right somehow.

  8. Anonymous Coward
    Anonymous Coward

    Yet...

    more and more companies, large and small are swallowing the MS Snake Oil and putting their faith in this Pile of steaming crud.

    I guess it will take a few major players to go Belly UP (bankrupt) because of extended outage on corporate email. Naturally MS is not responsible for any loss due to their cockups. EULA Page 78453 in 3pt font.

    Lookout is as apt as ever IMHO.

    1. Anonymous Coward
      Anonymous Coward

      Re: Yet...

      We (50k+ employee corporation, with offices worldwide) are part way through a Lotus Notes to Office 365 migration.

      Notes, while a little flaky occasionally, worked most of the time.

      I was one of the early migrations, about 5 months ago. We are about 80% done now, with a few stragglers mostly due to local account or client specific issues.

      We probably have a full outage (all O365 services) about every 2-3 weeks, and a partial (just Skype, or just Outlook, or just OneDrive etc) most weeks.

      As an example, we lost access to Exchange (365 hosted) from Thursday morning last week, and it took till mid day Friday for it to be working again, yesterday OneDrive for Business decided not to sync for several hours, and then asked for my account details later on before it would work again!

      This seems to be the 'norm' for Office 365! :-/

      1. Wzrd1

        Re: Yet...

        I suspect I know the company name, we just split from that company and are slowly migrating to Offal 365.

        Could replace the lot of either bloatware with a small bit of open source programs, ical, sendmail, jabber, etc.

        But, we'd have one problem with doing that - it would work.

    2. Terrance Brennan

      Re: Yet...

      I would say the issue is not so much with Office365 per se as with blindly putting their faith in a cloud solution which, as an earlier commentator pointed out, is a black box to the end users who have no control and are hoping whichever vendor they have entrusted their corporation to is halfway competent and really cares about them. The latter is improbable and the former unlikely. The cloud makes lots of money for the vendors based on scale so they want to get as big as possible whilst doing as little as possible. All of them.

  9. Mikel
    Windows

    They just aren't good

    Maybe they should sell coffee or something.

    Edit: Non sequitur - GOOGL is up over a half trillion USD.

    1. Hans 1 Silver badge

      Re: They just aren't good

      >Maybe they should sell coffee or something.

      Would you buy cold coffee?

      1. chivo243 Silver badge

        Re: They just aren't good

        Lots of people buy cold coffee from Fivebucks, I mean Starbucks. But from MS, not especially.

  10. steamnut

    Immersed in theirown pooh

    I think that these clowns are really demonstrating just how little testing they do.

    This is a basic problem that ought to have been found in testing.

    Also, why cannot they reverse the patch application until they fix it? Surely any software department would be able to do this and let the users continue to access the system?

    Yet another reason why I am never going to risk my systems by using Office 365 or Azure. If In go bust who do I sue?

    I think that Microsoft are discovering a new version of immersive experience - in their own sh*t! Smells pretty bad from here.

    1. Anonymous Coward
      Anonymous Coward

      Re: Immersed in theirown pooh

      Quote: "Also, why cannot they reverse the patch application until they fix it?"

      Playing devils advocate...

      Presumably the patching probably fixed something else, or added new functionality (otherwise why patch?).

      The case to back-out would have to weigh up the impact of removing the patch, which could be re-introduce an issue that it fixed, or removing functionality they (or someone) is now reliant on.

      There could also be dependences, back-out this patch, and 'n' number of other patches implemented at the same time also need to be backed out.

      Although this still doesn't excuse them for not testing properly in the first place, or implementing an updated fix by now.

  11. linicks
    FAIL

    To be honest

    I only use MS at work (have to), and if you just do a peruse through the windows registry, it is like the old fun thing that was around on the Internet years ago.

    *nix is a clean well wired system, each cable labelled, and knows what it does (and so does everyone else)

    MS, on the other hand, have BLOBS all put together (like the story here in El Reg with the Cat5 cabling messes), with hooks and crannies that nobody knows what they do (not even MS), so there's Bob Hope and no fucking HOPE when trying to do an upgrade (if that what it was).

  12. batfastad

    Exchange IMAP outage?

    7 day outage? More like 13 year outage!

    I don't know about anyone else but I have found IMAP under Exchange to be pretty much non-functional from at least 2003 when I first did a bit of Exchange wrangling.

    In fact it was the reason we canned M$ altogether at the time being a publishing company with a load of Macs and the Exchange connectivity options were Entourage if it worked, IMAP or the basic webmail.

    Mind you I don't begrudge anyone moving their Exchange into O365. The last thing I want to see these days is on-premise Exchange.

    1. Destroy All Monsters Silver badge
      Windows

      Re: Exchange IMAP outage?

      Mind you I don't begrudge anyone moving their Exchange into O365. The last thing I want to see these days is on-premise Exchange.

      This.

      Keeping fickle turdpiles in the server room is too burdensome (did I mention Microsoft Licensing Menacograms?) , so you move to the bigger fickle turdpile that somebody else manages and do a Hail Mary there won't be an turdavalanche.

      And who has written the admin interface to Exchange Online? It's really Babby's First Management Interface..

      1. Terrance Brennan

        Re: Exchange IMAP outage?

        I would argue that on-premises Exchange was a great piece of software up until Exchange 2013. I found it easy to install, configure, maintain, and use. My systems rarely had downtime and troubleshooting was relatively easy. However, since MS decided the cloud was where the money was on-premises Exchange has gone south. Exchange 2013 is written for Microsoft's use in their cloud and is not a good idea for a company to run themselves. MS wants all their users to move to their cloud and seem happy to make on-premises life miserable if need be. Then, when you get to the cloud this kind of crap happens.

  13. Fraggle850

    Yay! Go cloud!

    I've previously experienced problems with MS updates borking stuff but thanks to the power of cloud they can now do it on a grander scale. What's even scarier is that you're pretty much forced into win10 updates (unless you are enterprise grade IIRC) so good luck with that too.

  14. Anonymous Coward
    Anonymous Coward

    MS Release process

    Clean Compile - ship it!

    1. Rusty 1
      Happy

      Re: MS Release process

      Surely just no awkward and fiddly diagnostics like "the scope of this variable could be reduced".

      Some of those bigger and scarier messages like "is read before written", "unreachable" and "unresolved reference" are just too gnarly to wrangle with!

  15. andrewj

    Maybe they can fix its embarassingly slow performance while they're at it?

  16. Franco Silver badge

    Honestly didn't know it supported IMAP, haven't seen anyone using it since the days of the first iPhones before they got ActiveSync support. Comments above clearly indicate people do though.

    I use Office 365 for my business, purely because 1. It's free with my MAPS subscription and 2. Not having an Exchange Server just for me. Convenience outweighs my usual cloud scepticism.

  17. Tezfair
    FAIL

    I feel their pain (the engineers actually)

    I did an exchange 2016 update earlier this month which pretty much killed exchange and IIS. Despite restarting all the 'disabled during the update' services, the server remained borked. Fortunately I left the update until after a full backup which I then used to restore the VM.

    Not done the update yet, still testing the 'before update' and 'after update' VMs

    Back in December, KB3114409 munched it's way through a lot of my managed Outlook 2010's rendering them all 'safe mode', so who's in charge of testing the patches these days?

    1. 7layer

      Re: I feel their pain (the engineers actually)

      " so who's in charge of testing the patches these days?"

      Where they any at all like ever?

      Been having fun with M$ for a long time and dealing with Exchange as well, I could say it's always random, if the update will be installed successfully or not. Basically nothing could guaranty that it won't brake the server. At my new company in the last 3 years, Exch2010 got broken twice with M$ updates.

      So every roll-up update and service pack are a challenge...

      Since I touched the first Exchange I don't understand why it's so bloody complicate to hook up a DB server with a simple SMTP transport + web interface. Of course is much more complex than that, but still you got my point.

    2. Vince

      Re: I feel their pain (the engineers actually)

      Funny you should mention the testing.

      It is the view of everyone in my office that Microsoft has got shot of the entire QA team because they never used to be this bad - sure the odd slip up, but every month now there is some major problem with a patch for some product and a lot of them are fairly obvious scenarios to test for.

      1. Boothy

        Re: I feel their pain (the engineers actually)

        I wonder if they've implemented some form of CI?

        With automated testing, and not sorted out the issues before dismantling the test/QA team? So no 'net' to catch the issues, before they hit the real world!

  18. a_yank_lurker Silver badge

    SLA Liability?

    Does anyone on Orifice365 have an SLA? It looks like Slurp is guaranteeing about ~95% uptime.

    1. Hans 1 Silver badge

      Re: SLA Liability?

      No, they guarantee 100%, as in, at any given time, at least one of their services will be available ... Word 360, Excel 360, Outlook.com ... just don't expect all services to be available all the time ...

      We have had successive years of expired certificates, they finally managed to hire somebody with enough brains to renew them on time (third time^H^H^H^Hyear lucky?) and now this ...

      I used to use IMAP on Exchange 2010, however, every other week, the Windows service needed to be restarted as it became unresponsive. You could even render it unresponsive (as in crash) by moving a "lot of" mail around in your mailbox, like 1000 emails.

      IT got fed up and told me to use webmail instead ...

      1. Destroy All Monsters Silver badge

        Re: SLA Liability?

        Sadly, that sounds exactly like kmail + akonadi.

        With added benefit that akonadi will kill your machine after 24h uptime.

        Why no Quality Control, KDE?

      2. 7layer

        Re: SLA Liability?

        Oh yes, you got that feeling?!

        Imagine we got more than 100GB mail, but I'd say ours are pretty "good", because it stays stable for 3-4 months without any major hiccup and most of the users use IMAP.

      3. tempemeaty
        Big Brother

        Re: SLA Liability?

        Odd how these kind of failures force us into using the browser based means instead. My gut is telling me that failures like this are meant to. It's a way of making the world a more cloudy place by design.

        (just another worthless 2 cents of mine)

  19. Anonymous Coward
    Anonymous Coward

    when it cost a few dollars a year for you own IMAP/postfix/EXIM server why have office IMAP

    1. Anonymous Coward
      Anonymous Coward

      When you are on Linux an need to check your mail.

  20. Lee D Silver badge

    My employer's keep going on about Cloud. I keep pointing out that even Google and MS aren't immune to problems like this. Google status page keeps history, as well, so it's easy to prove.

    Stupid as it may sound, the maths actually favours running your own systems still. I'm a mathematician as well as an IT guy and I checked.

    In terms of complete outages of service, a basic redundant hardware setup with VM failover covers so much that you don't really need to put faith in such huge organisations. And there's still a weak link on the uplink to the Internet.

    The hilarious bit is really our email stats - in terms of collection of email from our Exchange server on a leased line, the downtime is currently measured in minutes per year. In terms of being able to send email via an external smarthost, we've already lost a week this year. Because their server got marked as a spamhost and our emails stopped going out. It took them days to fix it. And so we failed back to the ISP, who has limits that kicked in too early and basically rendered that useless too. So we sent directly from our leased line, and from a private external server, and were able to send without problems for several weeks.

    Basically, so long as you plan effectively, and have backup plans ready to go, you're actually safer than on the big Cloud plans that costs lots of money.

    And we looked into Cloud-based backup once. That was hilarious. It was a significant fraction of the entire IT budget every year to spend all month swamping our external lines to backup the barest of necessary data. Compared to a small annual expenditure on NAS, several units that can be taken offsite, and some decent backup software.

  21. James 51 Silver badge
    Childcatcher

    I am surprised that no one has mentioned that this is how they intend to run Windows 10. There could be millions of people with bricks in the corner in the future. Won't that be fun.

  22. CPU

    Have they tried turning it off and back on again?

    1. Anonymous Coward
      Anonymous Coward

      Yes, and it didn't turn back on.

Page:

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019