BlackBerry users should be getting the message again today – RIM reckons it has found and fixed the latest fault although there's still some backlog to be cleared. The company tells us that yesterday's outage (as opposed to Monday's down-time which remains unexplained) was down to a "core switch failure", and a failed back-up, …
Perhaps the real reason the servers fell over twice in such a short space of time isn't so innocent
Reading all these stories about BBM being down not one has ever mentioned the possibility of a hack. Different states have been asking RIM for access to their servers. Maybe one got impatient.
I thought i was the only one....
Has anyone actually thought RIM have been compromised? Why else would they be silent with no feedback to their customer base, surely if it was a simple hardware malfunction then they would have identified, fixed and provided a response, coupled with the fact i'm sure their DR plans would have kicked into play to save them the embarrassment of a lengthy outage. Drama over! However they need to address their total lack of communication... The silence is deafening and echoes the RSA debacle... They were breached and thousands of customers were left in the dark. RSA stayed quiet whilst it could get their PR teams in place with how best to deflect the heat and spin and also leave it long enough for people to find something new to talk about before releasing their official statement. They knew they'd been hacked the minute it happened and what they needed was time...
IF RIM simply had a server fail it wouldn't take down a whole network, i'm pretty sure the enterprise beast has more resillient DR strategies in place for this not to be effected... Which spurs my thoughts even more....
I'm intrigued to hear what Blackberry are going to do in order to redeem themselves from this one!
Blah, blah, blah....
"Core switch failure and a failed backup" says RIM...
What they actually mean is that they have failed to incorporate adequate resiliency in their infrastructure, and they have similarly failed to carry out appropriate disaster recovery testing that would have identified such issues.
Trying to find any information at all about RIM's resilience/contingency arrangements/promises on uptime is proving difficult. Anyone know anything about what they claim to have/promise to deliver?
Posted from my Blackberry
Still not fixed
I still can't send email on BIS or BES on my Orange BB. I did get a splutter of email this morning at 9am, but nothing since (and I'm seeing them in my Google mail). Some Facebook features are still borked too. Simply unacceptable for a service whose USP is messaging.
RIM what are you going to be compensating business users with after this lengthy and quite frankly diabolical outage?
I imagine there is a huge backlog of data on there network.
I can understand emails and BBM not coming through because of this but can someone explain why web browsing still doesnt work? I guess to much traffic on their network?
Either that or they are telling porkies and it's still broken.
I'm getting about a 2.5 hour delay at present on emails. I just got an email through on my blackberry that was in my inbox at 8:30 this morning (11:00 now). I sent myself a test mail at 9:00, still not got it on my blackberry yet.
It's not a huge impact for me today as I'm at my desk but it was a right pain on Monday when I was out and about.
My guess is
that the switches in question are managed by some MegaMouth Consulting business improvement partnership, who add value to the proposition by putting in PHBs all over the place to provide business continuity assurance within a integrated world-class best-of-breed process governance framework.
What will have happened is that PHB1 will have asked PHB2 to apply a scheduled upgrade, wiht a backout strategy in place.
PHB2 knows a bit about backing things out and in turn asks PHB3 to apply the upgrade and backout if necessary.
PHB3's knowledge of backing out is running for the pub toilet when he finds the gorgeous bird he's been chatting up is actually a transvestite. So it's something he prefers not to talk about in case his wife finds out.
So PHB3 tasks PHB4 with the upgrade, who in turns tells some downtrodden, overworked and untrained PFY to apply the upgrade. Do the core and redundant systems at the same time - delivery achieved quicker, time is money, young man.
So the PFY does his best, it all goes horribly wrong and it's taking days to fix. Guess who'll be fired for the disaster, though?
I knew something was wrong, when I tried to call my wife last night on my BB.
And the someone sounding like the person from the local takeaway asked me if I wanted egg fried rice with that.
Technically they don't have to provide any sort of SLA to there customers, atleast standard consumers. Businesses may be different. You guys need to remember that the Blackberry by its very nature is a business device and has only really become main stream in the past couple of years.
So what they lost there servers for a few days. I can't remember it ever happening before which aint bad considering how long they have been around. Give them a break jeez!
This is what you get when you have a management team that spend too much time patting themselves on the back for stuff they did 6 years ago and gazing adoringly into mirrors ever since.
bloody biz school grads
I bet you if you look over the last few years, RIM has probably massively increased it's marketing budgets (to attract the youfs) and amount it pays to shareholders. Same problems as Nokia. Use Business school grad rubbish like six Sigma and all those other processes instead of concentrating on being a TECH firm.
Business school grads shouldnt be let within 1000 miles of a tech firm. Don't worry about the product, the product doesn't matter, its ALL ABOUT THE BRAND. Thats fine for Nike and other trend products that will be forgotton within a month but for tech firms its unforgivable. Nokia AND RIM have forgotton that their products are critical products FIRST and a brand SECOND.
I don't care which celeb is using which phone, it's a phone first and if it looks nice, even better. The Playbook is an example of what is wrong with RIM. At some point, someone MUST have said "This is a piece of shit, we are an email company and this doesnt do that" and was either shot down by management or marketing gave the technies a date to meet for release and refused to budge it so an incomplete product was released.
That is what is worrying abotu Apple. Steve Jobs was able to shoot down ideas or push products and override marketing and release dates. Will the new boss do that?
How many technies do you know that will go to a Marketing Director and say "The produc isn't ready and won't be ready for yoru release date. Get back to your 2 hour drinkign lunches and we will give you a call when we want you"
Now i don't know this for certain obviously but i think they've been compromised why else wouldn't they have the network back up and running again...?
Young Jedi, much to learn you have
It is a common mistake to attribute an outage to nefarious hacking when simple human stupidity is a more probable cause.
RIM somewhere attributed the problem to a software upgrade gone bad. Having seen similar problems first hand in large (carrier) networks, the long outage doesn't surprise me. It ends up being like a self-inflicted DOS attack, and the more you try to fix it the more the problem cascades due to the law of unintended consequences.
But hey, it might be wily hackers after all.
I think the key point here is transparency and communication. Every company that provides a service will at times have problems with providing the service as advertised.
What is particularly bad about this is how poorly RIM are communicating with their customers. All the money being spent on building the brand through advertising is being unravelled by a failure to communicate.
At the very least I would expect them to have at least one person posting updates on their website and twitter feeds.
The two (count them! two!) updates posted so far on their website over the course of a three day outage are laughable for their lack of comprehension of the scale of the outage and the lack of information about what they are doing to remedy the outage.
From Phone Operators? From RIM?
Have a laugh.
Side note - Vodafone had a full days outage on their network and compensated no one (http://www.theregister.co.uk/2011/02/28/vodafone_outage_2/).
What did we do.....
how did we get ourselves here.......
isn't there more to life :-)
Many businesses have become reliant on mobile email and data services, this needs fixing, explaining, and if necassary re-engineering, and soon. Are RIM ever going to regain the trust of the business community?
The article is 7 hours old.
I still have no internet/email services with BB.
I suspect this will go into day four.
I'm not going to do the "it's the end of the world" thing or "I'm mad as hell"....
Better things to worry about...
Re: Nope sorry
We have a more recent article, http://www.theregister.co.uk/2011/10/12/rim_down_again/ .... it went up before lunchtime
problems here at work
Our BES connected Blackberries here at work are having data issues this morning. Emails and messages are not getting through. Phone works fine however.