Look on the bright side
At least they weren't holding it wrong.
Microsoft secure Azure Storage goes down WORLDWIDE
Microsoft's Windows Azure storage cloud is having worldwide problems with secure SSL storage, probably because Redmond let the HTTPS certificate expire. Being 'in the pink' is not good news for Windows Azure, as this screenshot from the Windows Azure Service Dashboard attests (click to enlarge) The problems were first …
-
Saturday 23rd February 2013 09:56 GMT cpreston
This doesn't kill "the cloud" for me
After commenting on a few other comments, I thought I'd make my own. First, I don't work for MS or any cloud vendor, so spare me the ad hominem attacks.
What I am is a huge user of cloud technology. In fact, my entire company runs in the cloud. Our phone system, our file sharing system, our CRM system, our payroll and accounting system, etc are all cloud providers.
Am I without outages? Absolutely not. Am I without problems? Nope. Have I fired some cloud vendors over the last few years? Yep.
But...
Do I have to hire and manage an IT staff? Nope. When I found out I didn't like a product, was I stuck with it because I paid for it up front and have to wait for it to depreciate? Nope. I just fired that cloud vendor and got another. Do I have to do any planning when I expand and contract the services I use? Nope.
I am far happier with my company's IT services than I was when I ran my last company and we did it all ourselves. And when there's a problem? I just call the guy. (2 1/2 men reference if that wasn't obvious.)
This is possibly the dumbest reason a cloud vendor has ever had an outage. But I have to quote the Big Man, "Let you who is without sin cast the first stone." I love all the comments that say they'll never trust the cloud because of things like this. As if you'd never have an outage if you did your own IT. Riiiiiigggghhht.
-
Saturday 23rd February 2013 13:20 GMT (AMPC) Anonymous and mostly paranoid coward
Re: This doesn't kill "the cloud" for me
Good points cpreston,
Microsoft Azure is not the only cloud service available. And I have never seen or heard of a data center that has met a 100 % SLA. Not in my past 25 years in the business. We will have to wait for Skynet Singularity and the deletion of all humans before that happens.
All these big cloud players must have at least one MAJOR fail on their books (look at AWS in Northern Virginia, FFS) as do .... *cough" .. most, if not all ,privately managed data centers.
Some of these massive failures make the news, and some ... don't.
Statistically, they are most often caused by human error.
Afterwards, failure can be measured economically (the right way) or by the degree of exposure and ensuing media frenzy (the most common way).
I will admit this is a rather SPECTACULAR fail because of the company, number of data centers, customers and zones involved. The apparent simplicity of the reason for which it happened just makes it juicier (but when is there ever a good reason?).
"Too big to fail" might work for banks and investment houses. But that 21st century oxymoron will never apply to data centers, networks, nuclear power plants, ATC systems, etc. Some things just can't be covered up. Hidden or ignored flaws have a way of finding you, usually late at night , when you are sleeping, on vacation or having a coffee break.
Whilst some of us can still rush in during that bank holiday weekend and bring the payroll system back to life before Monday morning (after a long-ignored, "too expensive to fix", SPOF finally pukes all over the system) and leave quietly with no one the wiser, that doesn't make us particularly smarter, it just makes us lucky.
Fortune 500 managed systems proposing 24/7/365 availability are obviously held to a higher standard. I suspect some highly-placed corporate dweeb-bonehead-ms-drone-bean-counter (choose all that apply) decided that automatic cert renewal might be too *expensive*. Which could also explain the magnitude of the policy error.
School boy mistake indeed.
Personally, I would call it a hanging offense, at the very least worthy of a hefty and vigorous b*tch slapping followed by 4 year re-assignment to an Alaskan data center. Pity the poor fool who ends up taking the rap.
Full disclosure: yes I did see this happen (once) in our shop, to a little-used web service.
We were lucky and smarter afterwards. Policy put in place, etc. forgiven and forgotten, bla
IMHO cloud computing will become a much healthier and more viable eco-system when it is no longer dominated by a handful of big players. Accidents like this will improve cloud deployments in the long run, particularly if companies start hiring and listening to talented people again.
Remember there is more than one way to do cloud, here are just a few:
1) Go public all the way, preferably with non-core services first until you gain confidence. There are many options out there, just go look at them.
2) Set up your own private cloud first and think about what you can peel off to a public cloud, when and if it becomes economically justifiable
3) Some hybrid of the two.
Of course, step 0 is to conclusively prove that any proposed cloud option(s) are less expensive and troublesome to run than by using your conventional data center. That is not always easy to do. But as soon as you have that proof, ît is time to to consider cloud services and solutions in the same way you consider any IT purchase. Of course, some research, assembly and homework will be required.
Remember that as long as it's cheaper and easier to do cloud options, companies will.
Although some companies may be CAREFULLY re-thinking that Azure subscription (or at least re-negotiating it).
Signed,
A self -confessed, mildly obsessed, fanboi, cloud-tech lover
-
Sunday 24th February 2013 01:27 GMT Anonymous Coward
Re: This doesn't kill "the cloud" for me
@AMPC
As someone who gets to pick the pieces up after such events as this, I can't help but disagree with all that you and cpreston have presented here.
I think cpreston summed up what cloud is, when he stated he didn't have to employ IT staff anymore. It's a non technical managers wet dream.
Now they get to sit in their plush offices doing less actual work (not managing them annoying technical people for example), pointing out to those who own the business how much money they are saving the company (by not paying for IT Staff, or equipment for example) whilst down playing the risks they are exposing the company to.
I think such people are deplorable, that they would be willing to gamble in some cases the very existence of a company, and all of the staffs jobs, so that they can have an easier life (not having those annoying technical people to deal with for example, no evaluations to do, no wages to pay, etc. etc)
The only possible way for any company to be able to ensure that data is secure, and available is to do it for themselves. That costs money, but it provides security for the existence of the company, and for the staff who rely upon the continued existence of the company for their jobs.
Anyone who is dumb enough to outsource the existence of their companies viability to some third party, deserves to fail. Any manager doing this for your company should be questioned very seriously about why they would want to expose your company to such risks.
-
Sunday 24th February 2013 08:27 GMT cpreston
Re: This doesn't kill "the cloud" for me
@Dave Dowell
I'm sorry you find me deplorable. I'm also sorry that you misunderstood my comment about hiring IT people. I am IT people with 20 years of experience in IT. I also now employ another IT person. But we choose to use our IT skills in different ways other than supporting the day to day IT operations of our small company (<10 employees). We have "outsourced" that to various other companies. I'm sorry if you think that means that I deserve to fail. I disagree.
My choices were a lot like other small companies:
1. Hire an IT generalist that will know a little bit about everything
2. Hire a bunch of specialists via cloud companies who know a lot about a few things
I chose #2, and I don't see how that makes me an idiot. I love how you think that people who are employed by your company are inherently less risky than cloud vendors paid by your company. It's a bias that I'm trying to expose.
Please tell me why having someone on my payroll makes them less likely to screw up than someone who works for a vendor that does nothing but the thing I hired them to do (e.g. run mail services for me).
A previous commenter mentioned about the lack of control you have in a cloud situation. That is true and it's not true. It's true in that you are one customer of thousands or millions, and you have little control over how they do IT. It's not true in that if you don't like how one cloud vendor does things, there are lots of other cloud vendors to take their place.
-
Sunday 24th February 2013 10:56 GMT Anonymous Coward
Re: This doesn't kill "the cloud" for me
@ cpreston
I'll let you in on a secret, the managers in the company you've out sourced your compute function to, do exactly what you do. They look around the market place and employ the cheapest resource they can get away with. Sure they could pay vast sums to employ proper specialists, but like you they have targets, and budgets, and so want their bottom line to look good, so that they can get their big bonus.
All them people you don't want to employ because you don't want to pay for their skill are just like the people running clouds. Except they don't know or care about your individual infrastructure. Your individual infrastructure is just one of many they run, it's no more personal to them than any of the other infrastructures they run. When it's in a heap on the floor, they're not completely dedicated to getting it back up, they're still making sure they deliver on the other infrastructures they run. They have targets to meet as well, and delivering on all them other infrastructures, even when yours in in a heap, is part of their targets.
Cloud has a place, indeed it can be a very useful resource, but any place you put it, which could result in your company not being able to survive in the event of a failure, isn't a place where cloud belongs. From your earlier statement about how you've put your entire company IT into cloud, that's what you've done. You've delegated the responsibility for your companies survival to some third party supplier. Who as I pointed out earlier is employing exactly the people you don't want to employ, but putting them under much more pressure to maintain lots of different infrastructures. They know less about your companies specific requirements, they know less about your companies specific needs, and they care less about both than anyone you employ who would be dedicated to understanding and caring about those things.
Still it's cheap, and you don't have to worry about planning, implementation and running costs. So who cares. The staff at your cloud providers certainly don't, they'll keep their jobs no matter how badly your company fails, they have other infrastructures to run after all, so their jobs are safe.
-
-
Sunday 24th February 2013 18:10 GMT (AMPC) Anonymous and mostly paranoid coward
Blue Pill? Red Pill?
Except that there is so much more to the "cloud" than Microsoft Azure.
First of all, I do not like using the term "cloud". It is basically a marketing buzz word and means whatever people want it to mean. Unfortunately, we need some kind of shorthand term so cloud it is, at least for today.
Second of all
Cloud != the end of IT as we know it.
Arguing that this Azure meltdown invalidates all cloud solutions is like saying the Toyota brake system recall invalidates the internal combustion engine.
Azure is a public "cloud" service that offers storage and compute resources for rent. For many companies, that can be incredibly useful and cost effective. Microsoft is one of multiple companies offering the same service. It is using technology that is readily available to private IT shops. This market is growing by 40 % annually.
Although Azure just failed at providing that service (for about 12 hours based on recent reports), it doesn't mean cloud services and technology can't be used successfully by people who know what their doing.
With the proper staff and know-how you can build your own Azure cloud and still save money. You may even do it better than MS (or not).
Like all tech, when used properly cloud solutions can save money and even generate new business.
But a LOT of people just can't or won't grok this.
I suspect that weavers and carriage drivers also refused to accept steam engines and cloth looms, but hey, whatcha gonna do?
Refusing to understand mega trends is a very counter-intuitive approach in a market where there is a massive shortage of people with cloud skills but clearly no shortage in ostrich techies.
A business needs IT systems that help them make and save money . This fundamental rule still applies to even the most "special" environment, particularly during a bust economy
You also imply that managers choosing cloud options have never thought about the risks. But I fail to see how managers can even assess these risks without some knowledge.
When was the last time a manager asked you about the risks associated with this or that redundant power supply, server clustering technology or the company's off-site backup strategy ?
Would "never", "once" or "hardly ever" be right answers?
Imagine if these same managers then asked their IT people about the cloud and were effectively told it "it is too risky and it sucks". Then next weekend, one of their country-club drinking buddies explained how cloud services saved their company 1,000,000 USD on last year's IT budget.
I suggest you at least learn about cloud options so you can explain them to PHBs, otherwise someone else will.
It really is that simple.
-
Monday 25th February 2013 01:12 GMT Anonymous Coward
Re: Blue Pill? Red Pill?
@ AMPC
Why is it cloud fanbois assume that you know nothing about cloud if you speak out about it being used badly?
You should avoid patronising people who express views you disagree with by implying they havn't got a clue what they're talking about, or sooner or later you might make the mistake of patronising someone who works in a cloud company.
Thank you for your advice, I am certain however that I understand cloud, so I shan't be rushing off to learn about it.
-
Tuesday 26th February 2013 11:37 GMT Vic
Re: Blue Pill? Red Pill?
Imagine if these same managers then asked their IT people about the cloud and were effectively told it "it is too risky and it sucks". Then next weekend, one of their country-club drinking buddies explained how cloud services saved their company 1,000,000 USD on last year's IT budget.
That's very easy. You just point out how much more cash they would have if they didn't bother to pay any insurance premiums for the year.
Cloud has its part to play in the future of IT, but I've yet to see an instance where the purported savings didn't come at the expense of increased exposure to risk.
Vic.
-
-
-
-
Sunday 24th February 2013 00:32 GMT Don Casey
Re: This doesn't kill "the cloud" for me
Cloud vendors are interested in your business (the money you pay them), not your Business (what you do to have people pay you money).
They don't know your wants/needs/tolerance for outages, etc. etc. If commodity is all you need from IT, then fine.
If IT is mission critical to you (and that is an ever-expanding universe) then you have to think twice about giving up even indirect control over what happens to the infrastructure running your Business.
Outsourcing is one thing; you have a direct line of communication to people who should understand your Business and can react accordingly when making changes and when things go sideways.
Going to The Cloud completely removes you from the decision makers and actors, you have no control over the how and when and all you can do is hope your business is enough to keep them competent enough so they don't screw up your Business.
-
Sunday 24th February 2013 00:33 GMT BinkyTheHorse
Re: This doesn't kill "the cloud" for me
Ah, so your ass *is* potentially on the line - I was wondering about that, reading yet another of your comments defending cloud services. You're not a cloud vendor, but you seem to imply that you're responsible for making technology decisions in your company, so if your current vendor has a similar titsup, someone can use this example as "you should have seen it coming" ammo against you. In any case, thanks for the disclosure.
The backlash against cloud services you see in the comments is really due to one simple promise: that switching to a cloud provider would effectively rid your IT infrastructure of single points of failure, modulo maybe a couple of minutes max downtime in the event of a very serious emergency. Problem, this is another cloud outage in the last couple of months with exactly the opposite characteristic.
In summary: I concede that cloud services are a Very Useful Thing, but the fact is they *have* been massively hyped and overadvertised. The current reaction is simply yet another IT silver bullet having a new one ripped.
-
Sunday 24th February 2013 11:07 GMT mmeier
Re: This doesn't kill "the cloud" for me
Yes, cloud service has been overhyped, even more when it comes to small/medium companies and cloud. You exchange one Single Point of Failure(SPF) like your server for another (your internet connection). You also have to trust your choosen cloud provider that he can deliver availability as contracted. BUT:
Data centers in the banks typically sell 99.99 (around 1h/year) or 99.9 (9h/year) to internal customers for most systems(1) so that should be enough for most companies even if you may be more affected by planned downtimes(2), And to get the 99.99 figure they use two centers on different locations, double data lines and all the stuff.
Outsourcing to reduce management jobs does not work. You exchange local IT where you can do the "800pf gorilla"(3) with remote IT where you threaten with contract clauses. Language differences get added even for those who speak the queens english as their primary language(4). If you do, you do it for money and likely for a non-IT business.
If not for the lousy internet speeds (and german laws) I could see the craftsman in my hometown doing it. Small/medium companies in germany use "cloud" already for tax and salaries with Datev (not called a cloud but technically the same) so doing it for bills and general writing would not be a new concept. And those companies could live with a 99 percent reliability since they have no time critical stuff
(1) Back in the early 2000 even stock trading was in that range
(2) That time from internal IT included planned downtime that where put "late" and not affect business
(3) A 190+ cm, still muscular if "manly girth" equiped man leaning on ITs desk does motivate many admins...
(4) Most Indian IT workes speak something that resembles english as much as Texan does (Or Sächsisch resembels German)
-
-
-
Saturday 23rd February 2013 12:08 GMT sysconfig
Everybody sees how M$ f***ed up yet again and thinks "the cloud is not an option". It's not *the* cloud. It's M$ who fucked up yet again in short succession, in their so-called cloud, which seems to be full of schoolboy errors and single points of failure.
The biggest SPOF sits at the helm of that company, still, for a reason which is beyond many people.
-
Tuesday 26th February 2013 11:42 GMT Vic
> It's not *the* cloud. It's M$ who fucked up yet again
Sure.
But the issue is about who cares..
If I screw up on customer site, the man infront of me cares. And that means I care.
If a cloud provider (possibly on a different continent) screws up, he's not nearly as motivated as I would be because he is remote from the problem. And the only people the end customer can talk to is a tech supoprt department who will know very little, have no ability to effect any fix, and probably won't even know who is trying to fix it.
Outsourcing your work to any remote provider means you will have less influence over them than you would if the job were done locally. Whether that makes a difference to the business case is something that needs to be managed and documented.
Vic.
-
-
Saturday 23rd February 2013 13:44 GMT Disgruntled of TW
O365 - backed by a financial guarantee ...
... yeah RIGHT. Microsoft promise to pay you the paltry amount you paid them for each user. The atrocious effect on your business is not their problem and never will be.
If you choose cloud, choose very, very carefully. Putting stuff there without understanding the risk to your business could be a career limiting move.
-
-
Saturday 23rd February 2013 21:04 GMT Cipher
Re: Data Utilities
Yeah, more Government intervention into the net, that's the ticket. It'll be freakin' perfect after the bureaucrats mangle it, censor it and tax it to death.
Remember when Clinton wanted copies of our encryption keys? Obama is writing Executive Orders right now to give him a "Internet Kill Switch" in the name of national security.
I don't know about where you live, but here in the USA, anything the Government touches turns into crap. Expensive, censored, politically correct crap...
-
Saturday 23rd February 2013 22:23 GMT Anonymous Coward
Re: Data Utilities
The day that happens is when a massive commercial advantage is handed over to countries that don't have a kill switch.
One day, some politicians will get the point, so I suggest you write your elected representatives and help them understand. Vote out the ones who don't.
,.
Otherwise prepare for Armageddon, at least on a political scale
-
-
Saturday 23rd February 2013 15:49 GMT JDX
Dearie me
Out of interest, have the other big players had similar screw-ups? I know things like gmail have gone down but has the entire google app cloud or EC2 flopped like this?
Maybe it's deliberate ploy by MS to make cloudy computing generally look bad so everyone stays with desktop computers, running Windows :)
-
Saturday 23rd February 2013 16:37 GMT Levente Szileszky
Once again I see my anti-MS-cloud stance completely justified...
...as I NEVER trusted this giant Ballmerian spider mess of services, not for a sec - remember the (fittingly named) B(ig)PoS? They had more downtime in one year, and always worldwide than everybody other hosted service provider (Google, Rackspace etc) TOGETHER, UNTIL DATE (years and years)...
...'oooh, Azure/WebOffice/O365/whatever-its-latest-newest-old-product is totally different, it's top-notch now' I heard every single time - and they just KEEP FAILIN' KEEP FAILIN'...
...so while I'm far from happy with Google's weird support I'm still enjoying my CEO's trust because I went with Google Apps Business and never had to deal with HOURS-LONG outages affecting our most critical communication//cloud services, thanks to Google.
-
-
Monday 25th February 2013 08:13 GMT amanfromMars 1
Do you Know what Tomorrow will Bring? Have you prepared Anything and/or Everything for Presentation
@AC 17:54 - I'm not here to win a BEAUTY contest, I am here to tell the TRUTH! And THEY CANNOT HANDLE THE TRUTH …. Eadon Posted Saturday 23rd February 2013 18:08 GMT
There is an added cloudy ingredient in the future azure blue skies thinking mix which is intelligently designed to render confusion and CHAOS to wannabe controllers and Dummy Operating Systems, Eadon, and that is that they don't know what the truth is to be.
This topic is closed for new posts.
Biting the hand that feeds IT
