Microsoft struggles against self-inflicted Office 365 IMAP outage Microsoft engineers are struggling to fix a seven-day-old, self-inflicted Office 365 IMAP outage. IMAP access to Office 365 tanked on January 18, meaning customers could not access emails using Exchange Online via IMAP or connect third-party mail clients via IMAP. Microsoft told disgruntled Office 365 customers that the …
Every new project I've been involved in is many, many layers divorced from the real software, OS and hardware that it runs on. Is it possible that there's so much high-level stuff involved in Azure that directly managing each O365 customer as a discrete Exchange instance isn't really possible anymore? They mention a failed update, but you would think an update to Exchange could be rolled back. Unless that update had a knock-on effect to other software-defined stuff for that customer, that is...
more and more companies, large and small are swallowing the MS Snake Oil and putting their faith in this Pile of steaming crud.
I guess it will take a few major players to go Belly UP (bankrupt) because of extended outage on corporate email. Naturally MS is not responsible for any loss due to their cockups. EULA Page 78453 in 3pt font.
Lookout is as apt as ever IMHO.
We (50k+ employee corporation, with offices worldwide) are part way through a Lotus Notes to Office 365 migration.
Notes, while a little flaky occasionally, worked most of the time.
I was one of the early migrations, about 5 months ago. We are about 80% done now, with a few stragglers mostly due to local account or client specific issues.
We probably have a full outage (all O365 services) about every 2-3 weeks, and a partial (just Skype, or just Outlook, or just OneDrive etc) most weeks.
As an example, we lost access to Exchange (365 hosted) from Thursday morning last week, and it took till mid day Friday for it to be working again, yesterday OneDrive for Business decided not to sync for several hours, and then asked for my account details later on before it would work again!
This seems to be the 'norm' for Office 365! :-/
I would say the issue is not so much with Office365 per se as with blindly putting their faith in a cloud solution which, as an earlier commentator pointed out, is a black box to the end users who have no control and are hoping whichever vendor they have entrusted their corporation to is halfway competent and really cares about them. The latter is improbable and the former unlikely. The cloud makes lots of money for the vendors based on scale so they want to get as big as possible whilst doing as little as possible. All of them.
I think that these clowns are really demonstrating just how little testing they do.
This is a basic problem that ought to have been found in testing.
Also, why cannot they reverse the patch application until they fix it? Surely any software department would be able to do this and let the users continue to access the system?
Yet another reason why I am never going to risk my systems by using Office 365 or Azure. If In go bust who do I sue?
I think that Microsoft are discovering a new version of immersive experience - in their own sh*t! Smells pretty bad from here.
Quote: "Also, why cannot they reverse the patch application until they fix it?"
Playing devils advocate...
Presumably the patching probably fixed something else, or added new functionality (otherwise why patch?).
The case to back-out would have to weigh up the impact of removing the patch, which could be re-introduce an issue that it fixed, or removing functionality they (or someone) is now reliant on.
There could also be dependences, back-out this patch, and 'n' number of other patches implemented at the same time also need to be backed out.
Although this still doesn't excuse them for not testing properly in the first place, or implementing an updated fix by now.
I only use MS at work (have to), and if you just do a peruse through the windows registry, it is like the old fun thing that was around on the Internet years ago.
*nix is a clean well wired system, each cable labelled, and knows what it does (and so does everyone else)
MS, on the other hand, have BLOBS all put together (like the story here in El Reg with the Cat5 cabling messes), with hooks and crannies that nobody knows what they do (not even MS), so there's Bob Hope and no fucking HOPE when trying to do an upgrade (if that what it was).
7 day outage? More like 13 year outage!
I don't know about anyone else but I have found IMAP under Exchange to be pretty much non-functional from at least 2003 when I first did a bit of Exchange wrangling.
In fact it was the reason we canned M$ altogether at the time being a publishing company with a load of Macs and the Exchange connectivity options were Entourage if it worked, IMAP or the basic webmail.
Mind you I don't begrudge anyone moving their Exchange into O365. The last thing I want to see these days is on-premise Exchange.
Mind you I don't begrudge anyone moving their Exchange into O365. The last thing I want to see these days is on-premise Exchange.
This.
Keeping fickle turdpiles in the server room is too burdensome (did I mention Microsoft Licensing Menacograms?) , so you move to the bigger fickle turdpile that somebody else manages and do a Hail Mary there won't be an turdavalanche.
And who has written the admin interface to Exchange Online? It's really Babby's First Management Interface..
I would argue that on-premises Exchange was a great piece of software up until Exchange 2013. I found it easy to install, configure, maintain, and use. My systems rarely had downtime and troubleshooting was relatively easy. However, since MS decided the cloud was where the money was on-premises Exchange has gone south. Exchange 2013 is written for Microsoft's use in their cloud and is not a good idea for a company to run themselves. MS wants all their users to move to their cloud and seem happy to make on-premises life miserable if need be. Then, when you get to the cloud this kind of crap happens.
I've previously experienced problems with MS updates borking stuff but thanks to the power of cloud they can now do it on a grander scale. What's even scarier is that you're pretty much forced into win10 updates (unless you are enterprise grade IIRC) so good luck with that too.
Honestly didn't know it supported IMAP, haven't seen anyone using it since the days of the first iPhones before they got ActiveSync support. Comments above clearly indicate people do though.
I use Office 365 for my business, purely because 1. It's free with my MAPS subscription and 2. Not having an Exchange Server just for me. Convenience outweighs my usual cloud scepticism.
I did an exchange 2016 update earlier this month which pretty much killed exchange and IIS. Despite restarting all the 'disabled during the update' services, the server remained borked. Fortunately I left the update until after a full backup which I then used to restore the VM.
Not done the update yet, still testing the 'before update' and 'after update' VMs
Back in December, KB3114409 munched it's way through a lot of my managed Outlook 2010's rendering them all 'safe mode', so who's in charge of testing the patches these days?
" so who's in charge of testing the patches these days?"
Where they any at all like ever?
Been having fun with M$ for a long time and dealing with Exchange as well, I could say it's always random, if the update will be installed successfully or not. Basically nothing could guaranty that it won't brake the server. At my new company in the last 3 years, Exch2010 got broken twice with M$ updates.
So every roll-up update and service pack are a challenge...
Since I touched the first Exchange I don't understand why it's so bloody complicate to hook up a DB server with a simple SMTP transport + web interface. Of course is much more complex than that, but still you got my point.
Funny you should mention the testing.
It is the view of everyone in my office that Microsoft has got shot of the entire QA team because they never used to be this bad - sure the odd slip up, but every month now there is some major problem with a patch for some product and a lot of them are fairly obvious scenarios to test for.
No, they guarantee 100%, as in, at any given time, at least one of their services will be available ... Word 360, Excel 360, Outlook.com ... just don't expect all services to be available all the time ...
We have had successive years of expired certificates, they finally managed to hire somebody with enough brains to renew them on time (third time^H^H^H^Hyear lucky?) and now this ...
I used to use IMAP on Exchange 2010, however, every other week, the Windows service needed to be restarted as it became unresponsive. You could even render it unresponsive (as in crash) by moving a "lot of" mail around in your mailbox, like 1000 emails.
IT got fed up and told me to use webmail instead ...
My employer's keep going on about Cloud. I keep pointing out that even Google and MS aren't immune to problems like this. Google status page keeps history, as well, so it's easy to prove.
Stupid as it may sound, the maths actually favours running your own systems still. I'm a mathematician as well as an IT guy and I checked.
In terms of complete outages of service, a basic redundant hardware setup with VM failover covers so much that you don't really need to put faith in such huge organisations. And there's still a weak link on the uplink to the Internet.
The hilarious bit is really our email stats - in terms of collection of email from our Exchange server on a leased line, the downtime is currently measured in minutes per year. In terms of being able to send email via an external smarthost, we've already lost a week this year. Because their server got marked as a spamhost and our emails stopped going out. It took them days to fix it. And so we failed back to the ISP, who has limits that kicked in too early and basically rendered that useless too. So we sent directly from our leased line, and from a private external server, and were able to send without problems for several weeks.
Basically, so long as you plan effectively, and have backup plans ready to go, you're actually safer than on the big Cloud plans that costs lots of money.
And we looked into Cloud-based backup once. That was hilarious. It was a significant fraction of the entire IT budget every year to spend all month swamping our external lines to backup the barest of necessary data. Compared to a small annual expenditure on NAS, several units that can be taken offsite, and some decent backup software.
This post has been deleted by its author
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
