O2's titsup network struggles to find its feet O2's mobile network is struggling to its feet after a 19-hour outage left thousands of customers unable to make or receive calls. This morning 2G should be working again but 3G remains patchy. That's according to the sluggish status page, which, between apologies, claims the 3G service is slowly being revived. The telco …
There aren't very many - lots of mergers, lots of companies who've gone to the wall. The only way those wafer thin margins can work for you is to have scale. Look at the telecoms companies around a decade ago, look at how many are still around. Sometimes the brand survives even if the company behind it has gone.
Or, the beginning of a new trend, one where the manglement ID10Ts actually have to face some real consequences for their fuck ups. And, I don't mean leaving with a Golden Parachute either.
IMHO, the consequences for those ID10Ts at RBS for their outsourcing decisions should be immediate termination, no severance pay or benefits (IOW a SKITA {Swift Kick In The Ass!!!})
If any of them squawk about it, then sue them for breach of fiduciary duty, and attach any, and all of their possessions to collect on a judgment. It is about time to start demanding that DAMAGEMENT be held accountable for its colossal fuck ups.
This bullshit has gone on long enough!
"For most big publicly listed companies, the old style belt & braces & and a spare pair of trousers approach to vital big-tech doesn't happen anymore."
That's because people aren't prepared to pay the prices that such an approach would result in. Almost everyone buys on price - the cheaper the better. The choice providers face is to take out cost and remain competitive - but less resilient - or to stick with the belts and braces, lose customers and go bust.
I disagree. Companies do spend a lot of money and effort building systems and services with "resilience" built in.
What they often don't appreciate is how much complexity this "resilience" adds, and that the complexity itself results in additional failure modes (which are harder to diagnose).
It's pretty easy to design a system which is resilient against a node powering off completely. It's much harder to design a system which is resilient against a node which starts sending out corrupt data. Google for "byzantine fault tolerance".
http://en.wikipedia.org/wiki/Byzantine_fault_tolerance#Byzantine_failures
"For example, in 2008 Amazon S3 was brought down for several hours when a single-bit hardware error propagated through the system.[2]"
With core infrastructure of large companies (RBS payment processing, 02 Telephony), it should not be possible to nuke the system, a suffcient amount of money and resources can easily render this possible.
Unfortunately the senior managers whoose bonus is based on cutting easily visible costs, and quite frankly have naff all ability at risk management, will cut these services to get their bonus, and if they are still around when the risk occurs, (a) they probably won't lose their job, and will (b) keep their bonus, despite how much money they have just lost their company(*)
This is why Barclay/Libor is a side show, it's being hanfled by the regulators, what needs serious consideration by parliment is the amount of the UK economy that is bet on technology services, that is not robust, and does not have an adequate BC/DR plan. RBS took out around a 25% of the abilitty of the UK to transact business, it is not unusual from other firms (e.g. 02) in it's approach to IT.
The modern economy needs legislation, that makes your IT and BC/DR as well audited as your accounts, shareholders and business partners should be able to read your company report, and see if they want to do business with you.
Completely agree. The key to it all is the evil tower of accountants. In virtually all companies the provision of IT services is seen as ONLY a cost centre, the imbeciles counting their magic beans try and cut funds to any department that is not a Profit Centre (except their own in most cases .. holders of the golden keys) ... This combined with management who don't really know their business creates this mess ... Whilst there is little to ague with the desire to run any department efficiently, the constant erosion of investment leads ultimately to failure...
The dimwits at the top who have got there through lies and self promotion need to wake up and see that without a good , reliable IT infrastructure their businesses will not survive ... Going for the cheapest option on IT eventually leads to reduced income/profi in the long term
It's a horrible thought and nobody's mentioned it, so I thought I'd better at least ask the question.
As I understand it, O2 have no idea what caused the outage. And I believe they are the network provider (via their "Airwave" service) for the UK Emergency Services.
Could this have been a trial run by someone who wanted to see what the effects on the Emergency Services communications would be? And do we know if they were affected at all?
Sounds a tiny bit like the plot from an Oceans X film... Take down the airwave or take the manager of that bank (Casino etc) off the air and then they can't call for help.
On the subject of call for help - what about someone who couldn't call for an ambulance or the police... Would they have a case ? or are the T&C's sufficiently wide when you take a mobile contract that paucity of coverage / service is not covered...
But then do they have a Duty of Care to their customers ?
Would it be possibly during a court case to demand the details of what did happen (same goes for RBSwest)
Whilst I can't disagree with most of the content I do have to disagree with the inferred presumption of choice. Whenever I have taken a service from a Mobile Phone company as a consumer I have never been asked if I want a best effort or guaranteed service ... therefore the point is mute.
Similarly mobile services purchased by large organisations, who spend vast amounts on building stable , redundant IT infrastructure, use the very same service ... I know which they would choose if there were different services offered.
If the TelCo's infrastructure has been built less in a less stable or redundant manner , that is purely their choice. It's a business choice, drive down costs and accept the possibility of service loss and disaffected customers , or not ...
"HLRs are not, usually, built in redundant pairs, which means the links are not either."
Back when the operational support systems in a telco were designed by engineers rather than Windows weenies from BSS or IT, HLRs often didn't need to be built in redundant pairs because they were built on fault tolerant Tandem hardware (or occasionally on VMSclusters), where the resilience is (mostly) provided by the supplied computer system. They would still need resilient connectivity, but if a telco doesn't have resilient connectivity for that kind of thing, then...
That was a decade or more ago though. I'm out of touch with what is currently fashionable (aka "best practice"?) in this particular sector of where IT meets proper engineering. Anyone got current info?
The Register 
Biting the hand that feeds IT
Copyright. All rights reserved © 1998–2024
