back to article Microsoft secure Azure Storage goes down WORLDWIDE

Microsoft's Windows Azure storage cloud is having worldwide problems with secure SSL storage, probably because Redmond let the HTTPS certificate expire. Being 'in the pink' is not good news for Windows Azure, as this screenshot from the Windows Azure Service Dashboard attests (click to enlarge) The problems were first …

COMMENTS

This topic is closed for new posts.
  1. stanimir

    Not the 1st SSL accident by Microsoft, it happened (iirc 2000) once w/ hotmail / password. Does anyone have fresher memory?

    1. Peter2 Silver badge
      Meh

      That was my first thought.

      http://news.cnet.com/Good-Samaritan-squashes-Hotmail-lapse/2100-1023_3-234907.html

      You'd think that a company the size of Microsoft could afford to run a system with better uptime during designated working hours than the majority of us lot manage to deliver to our businesses. Then again, size is probably the problem. I would imagine that Microsoft is one of those tremendously process managed places and "it wasn't my job" can now be heard echoing from a thousand voices crying out in terror.

      1. Anonymous Coward
        Joke

        @Peter2

        "I would imagine that Microsoft is one of those tremendously process managed places and "it wasn't my job" can now be heard echoing from a thousand voices crying out in terror."

        Lets not be too hard on them; I for one wouldn't be surprised if Win8 also had some influence in all this.

        You see; when Outlook starts on my PC it pops up a window with a list of chores to keep in mind (todo items), which I then work on during the day. I can well imagine that in the full-screen Win8 interface such a pop-up could end up somewhere in the background, thus easily missed and never looked at.

    2. jonathanb Silver badge

      From memory, they forgot to renew the domain in that case.

    3. Levente Szileszky

      "we are so sorry"

      http://forums.theregister.co.uk/forum/1/2013/01/04/windows_media_center_epg_fixed/

  2. Anonymous Coward
    Coffee/keyboard

    >"It is the opinion of The Register that to have a core service fail in every data center across the world simultaneously is an extremely bad thing to happen to a cloud provider."

    Now that was funny!

  3. Dazed and Confused

    Proves its secure

    So secure we won't let anyone access it

  4. Bob Vistakin
    Facepalm

    For crying out loud! How many times????

    This is a whole different kind of incompetence for which a word has not yet been invented. Seriously - this is the same me-too comedy "Cloud" offering from Microsoft which fell over last time because it didn't know about leap years? So hows that "lessons learned" thing coming on then?

  5. Anonymous Coward
    Anonymous Coward

    single point of failure

    there must be an attack or two to be developed here? enquiring minds want to know.

    1. Captain DaFt

      Re: For crying out loud! How many times????

      I hereby propose (and apologise to Matt Groening), that such stupidity be called a "ballmer". *

      1. eulampios

        @Captain DaFt

        Nice one,

        I'd even suggest three forms of this concept: bummer, balmer (pronounced in the British way: bɑ:mer) and ballmer

        --What a bummer, I forgot the keys in the car!

        -- It's Feb 29th, and who'd ever have predicted that, what a balmer?!

        -- Your Azure is down? Don't worry it's a planned ballmer!!!???

      2. Levente Szileszky
        Stop

        Re: For crying out loud! How many times????

        Err, I'm sorry, I claim MY COPYRIGHT as I'm ALREADY USING "BALLMERIAN" as adjective as well as a noun, look up my posting history. Also just recently started using it as a verb (on another forum) eg. ballmering up = utterly, royally ****ng up something that supposed to be trivial...

        ...but hereby I grant a royalty-free 'use as is' license to ALL ElReg posters on this forum, effective immediately, until revocation.

    2. Anonymous Coward
      Facepalm

      Re: single point of failure

      >"there must be an attack or two to be developed here? enquiring minds want to know."

      Two attacks, in fact: username = "sballmer" or "bgates", password = "password".

  6. Anonymous Coward
    Windows

    Coincedence?

    Now, obviously the SSL expiration is a plain out schoolboy mistake, anyone could have foreseen that. Still; in our small Dutch country those kinds of stupidities even happen at government level.

    But the other aspect is intriguing, especially if you take in mind that Microsoft has also recently released version 4.5 of their Web platform installer. Highlights in this version 'spotlight' are fully aimed at Azure; "Azure SDK for PHP", "Azure SDK for Node.js", "Windows Azure PowerShell" and obviously at first place the Azure SDK for .NET.

    Those .NET SDK's are for both VS2012 and 2010, both dated at Februari 2013.

    Heck; they even added a new software section: "Windows Azure", even though one could argue that Azure is basically another framework and as such should be listed as such (under 'framework' you'll also find stuff such as Python, PHP, obviously the .NET framework, Node.js and so on...).

    Could it be that... ?

    Microsoft is really pushing Azure as of late. And it seems either their 'success' or something else has now come to haunt them. And if you look at the prices which they charge then I think some customers have every right to be upset about all this.

    (for those of you unknown to the 'web platform installer': its a sort of package manager for Windows which allows you to quickly install specific Windows products. From IIS to SQL Server Express right down to MySQL, PHP, Python, Silverlight and their Visual Studio express versions. If you're interested in (web) development on Windows then this tool is IMO the best & easiest place to start looking).

    1. asdf Silver badge

      Re: Coincedence?

      >Now, obviously the SSL expiration is a plain out schoolboy mistake, anyone could have foreseen that. Still; in our small Dutch country those kinds of stupidities even happen at government level.

      Wow in most places these kind of things are expected out of governments but not so much multinational enterprises.

      1. Denarius
        Meh

        Re: Coincidence?

        I take it you have not worked at a multinational ? In IT anyway. Mostly starting with the well known phrase, "We have a standard process, what could possibly go wrong ?" Another instance; quote "A lot of good local decisions could be bad for the company"

        1. AndrueC Silver badge
          Happy

          Re: Coincidence?

          I take it you have not worked at a multinational ?

          Well yes, fair point. However it's the OP's apparent faith in governments that is so shocking.

      2. Anonymous Coward
        Anonymous Coward

        @asdf

        "Wow in most places these kind of things are expected out of governments but not so much multinational enterprises."

        Governments usually rely on (Enterprise-like) companies to handle the IT setup for them.

        Well, except for the time when someone within the government really felt good about DigiNotar I suppose ;-)

    2. itzman

      Re: Coincidence?

      Oh dear.

      I remember one of our engineers being asked to do a security audit for a large company.

      The firewall was fine.

      The 37 private modems attached to DDI lines allowing staff to phone in from home were not.

      When asked what would be the easiest way to steal information he replied 'Id simply attack, or bribe the junior member of staff who takes the nightly tape backup off site every night in her handbag'

      Of course these days, you just nick the laptop the sales director carries with him everywhere with a full copy of all the corporate data on it.

      Or better still, stick a trojan horse on it.

  7. Gert Leboski
    Linux

    Hahaha

    Mwuhuhuhahahahaaaaa!!!!

    That is all. :)

    Couldn't have happened to a nicer company.

    1. Anonymous Coward
      Linux

      Re: Hahaha + Hahaha = Double Ha Ha.....

      To bad the entire Microsoft ecosystem and all the data on it, didn't get wiped out in one huge fucking unrecoverable corrupted erasure as well.

      Ooooooooooooooooo that wooda stung.

      1. Mikel

        Re: Hahaha + Hahaha = Double Ha Ha.....

        Google "Microsoft danger data loss".

  8. Tel

    Oh Rly?

    And they want us to subscribe to Office 365 and store all our documents that run our businesses in the cloud if we don't want to pay through the nose for a licence to use only once and then throw away a physical local install of Office?

    Just who do Microsoft think they are and just how f***ed up is their business model these days?

    I've *never* trusted cloud computing and I never will. At least I can get at the culprits if my network goes down.

    How did Microsoft get this big of a deathwish without anybody in their corporate structure noticing?

    1. Fatman Silver badge

      Re: Oh Rly?...At least I can get at the culprits if my network goes down.

      Which is the attitude of my boss. She know exactly whose ass must be properly motivated1 to insure that the network functions. She also know whose ass needs to have a fire light under it, in the event network services goes down. Rarely has she ever needed to even threaten to get out the flame thrower. The network has issues, they get fixed. End of story.

      1 Includes purchases of the latest shiny-shiny to keep motivation high. It also includes the requisite monetary compensation that keeps acquired knowledge in-house, and deters pilfering poaching by other employers.

      1. Steady Eddy

        Re: Oh Rly?...At least I can get at the culprits if my network goes down.

        Fatman, where do you work, and how to I apply?

    2. AndrueC Silver badge
      Joke

      Re: Oh Rly?

      I've *never* trusted cloud computing and I never will

      Admiral Kirk once said that about the Klingons. Mind you a few years earlier he also said "Double dumbass on you" and that applies here as well :)

    3. cpreston
      Thumb Down

      Re: Oh Rly?

      Yes, because having all of your IT under your direct control always guarantees that you will never have an outage. That's why no company in the world ever reported an outage until cloud computing came around.

      But I probably shouldn't bother you with facts. You're mind is obviously already made up.

      1. Joe Montana
        WTF?

        Re: Oh Rly?

        It's not about guaranteeing you won't have an outage, that is impossible...

        It's about knowing what you have, how its setup, and if something fails what's gone wrong and what you can do about it.

        It's all about knowledge, control and understanding risk. With a third party cloud provider you have no idea how well (or otherwise) setup their infrastructure is, how resilient their hardware and power is, how it all fits together and you have no ability to fix anything if it breaks. With your own infrastructure you do know all these things, and you can make your own decisions according to how much resiliency you need vs available budget.

      2. itzman

        Re: Oh Rly?

        You miss the point.

        I ran a small ISP once. I had a customer escalated to me screaming that the mail was down and no one would tell him for how long.

        I asked why he needed to know that. 'Because I am trying to meet a legal deadline for filing some changes to a contract: I need to get the stuff there in 2 hours, if I cant email it I can fax it or get a courier'

        I said 'two hours I cannot guarantee'. THANK YOU he said. I will fax it instead.

        The moral is its not the outage that's the problem. Its knowing how long and what is wrong that allows you to take sensible steps to minimise the impact. That is FAR more available when its YOUR IT department who have the server in bits on the floor peering at the smoking CPU chip..

        a third party company simply doesn't want to tell you.

    4. Stewart Cunningham

      Re: Oh Rly?

      I think you'll find that it is now Office 364, possibly soon to be renamed Office 363

  9. adnim Silver badge
    Meh

    Some people...

    haven't noticed.

    If it wasn't for fsckups, pointless devices and trying to convince serious computer users that a finger is the most accurate input device. I would have forgotten Microsoft existed,

  10. Gray
    Trollface

    Oh, mercy! So now the process goes: a disruption is a "tits-up," a calamity is a "f***k-up," and now a total meltdown is a "ballmer!" .... oh, Mother, spare us!

    Thank you, Captain DaFt! "I hereby propose (and apologise to Matt Groening), that such stupidity be called a "ballmer".

  11. Numpty

    Xbox Cloud Storage is definitely down as well (and presumably related) -- http://support.xbox.com/en-GB/xbox-live-status

  12. This post has been deleted by a moderator

    1. ChrisM

      Re: This is HILARIOUS! AZURE FAIL.

      There is plenty of fail to go round...hopefully the biggest casualty will be the idea that you csn trust the cloud.

      The unseen cascading failure with this and any other cloud outage is the SLA's that the end users have with their clients. If I have control then I have oversight and more importantly responsibility and that is what is lacking when all your stuff is 'in the cloud'

      1. cpreston
        Thumb Down

        Re: This is HILARIOUS! AZURE FAIL.

        And you, nor anyone that works for your company would ever make a mistake that would cause an outage. EVER.

        1. Anonymous Coward
          Windows

          @cpreston

          That's not the issue, you're now ignoring the key fundamentals on which cloud computing was based: redundancy; the option to provide a service which can be spread across several (hardware) components. Thus if one component fails then others take over and there's no loss of continuance.

          At least that's the theory. Even so; it shouldn't be that hard to (automatically) move virtual instances from one setup to a backup in disasters like these. Yet even that doesn't seem to happen.

          So quite frankly I don't think people are uberly negative; they're merely demanding that Microsoft makes good on its promises.

      2. Richard 26

        Re: This is HILARIOUS! AZURE FAIL.

        "There is plenty of fail to go round...hopefully the biggest casualty will be the idea that you can trust the cloud."

        I've not used Azure but I find it interesting that (apparently) not a single one of the end users has a check that they are using a certificate that is about to expire. Kudos to Brian Reischl but he would have got bonus points for pointing it out yesterday.

      3. Anonymous Coward
        Anonymous Coward

        Re: This is HILARIOUS! AZURE FAIL.

        I think it's fair to say that any company who puts all of their eggs in one basket is doomed to fail and fail big - such as sticking all of your data in the "cloud" and hoping for the best. Used wisely, the cloud can provide effective scalability at a known price.

        Don't expect it to include an magic wand to instantly protect all of your data and never go down.

        If the data is mission critical and you can't afford an outage of a few hours, this isn't the solution you're looking for.

    2. Anonymous Coward
      Anonymous Coward

      Re: This is HILARIOUS! AZURE FAIL.

      Gloating is a sign of insecurity Eadon. Stop it.

      1. This post has been deleted by a moderator

        1. Anonymous Coward
          Anonymous Coward

          Re: This is HILARIOUS! AZURE FAIL.

          @Eadon - Even for you that's beyond pathetically childish.

  13. Robert E A Harvey

    Altered mind

    I entered the story thinking 'shit happens' and being quite sympathetic to the microsofters.

    Then I saw the first paragraph. " let the HTTPS certificate expire."

    Tossers.

  14. amanfromMars 1 Silver badge

    Global Control Systems Meltdown .....?*

    Are the world's brainwashed masses, ignorant of ITs novel virtual abilities at extraordinarily rendering colossal change, and in the new worlds emerging, strident and confident and disruptive, and also in cyberspace, being misinformed and diverted in thought to believe that China is an enemy and anonymised individual hackers, ..... invariably subjects of establishment entrapment whenever objects of APT prosecution and persecution, both adept and inept, actively engaging and thoroughly concentrated on phantoms and ghosts of sacrifice, .... are responsible for Microsoft's mistakes, and compounding code errors, in securing their operating systems as pimped and pumped to governments as a means with various platforms with which to control realities deemed suitable for media presentation with an invisible hand control ........ http://www.bbc.co.uk/news/technology-21556611

    Methinks the wheels have fallen off that broadband wwwagon before it even gets to the starting line in the New and Surreal IntelAIgents Space Race, don't you?

    *And only shared as a question for those who are unable to believe what can nowadays be easily done by remote powers with virtual control of true hearts and smarter minds.

    Note to Patten and the BBC .... Get your FCUKing act together, please, or clear the decks of dead wood and old hacks. GCHQ expects and all that jazz and razzamatazz ...... http://en.wikipedia.org/wiki/England_expects_that_every_man_will_do_his_duty

  15. Novex
    FAIL

    There are so many reasons...

    ...why cloud isn't a good idea that outweigh its one or two plus points. This kind of balls up is one of them.

    Some of the others:

    Lack of guaranteed privacy (just who else really can see my data, and that of my contacts and my calendar?)

    Lack of truly robust security (there's no such thing, all it takes is one disgruntled insider and my data along with god knows how many others' is in a hacker's hands)

    Reliability (not just of the cloud system, but the internet to it - what if my ISP has a problem, or my physical internet connection is in some way down?)

    The only good use I can personally think of is as a place to put strongly encrypted backup files.

    1. cpreston

      Re: There are so many reasons...

      1. If you think you have privacy of your contacts & email within your company, I think you'd be surprised with the reality.

      2. You know absolutely nothing about what "the cloud" offers as security, do you? I've seen some of the offerings and was more impressed with that they offered than what I see in typical shops.

      3. If your ISP is down, your business is down. Cloud or no cloud. But to your greater point of reliablity, I again say that just because something is in your datacenter doesn't make it more reliable.

      Amen on encryption. But I have to disagree that it's the only thing the cloud is good for.

      1. Paul Crawford Silver badge

        Re: There are so many reasons...

        I think for (1) they mean outside of their company.

        For (2) please provide more, most of the stuff I have seen mentioned is only secure between your PC and the servers, not 'secure' on their servers in the sense of encrypted by a key they do not hold. Also when it comes to encryption I would ONLY trust an open source implementation of the client side, otherwise how can you check at all if it has a back door? (Yes, I know there could be subtle flaws introduced to make GCHQ/NSA/etc's job easier, but its about the best you can ask for).

        (3) depends on your business, for some work yes it is but for a lot of design and manufacturing work you can work quite well (in some cases better!) without an internet connection for a day or so.

      2. Novex

        Re: There are so many reasons...

        1. I'm the only person in my company, so I suppose I could leak my info, but I'd have to sack myself if I did so...

        2. The cloud only offers as much security as 'they' say it does - we don't actually know for sure how each provider of cloud-type services ensures that data doesn't get into the wrong hands. It's certainly true that it's in the providers' interests not to treat users' data badly, but that doesn't mean things can't go wrong. And as far as I'm concerned, better the dickhead I know in my company than the dickhead I don't know in someone else's.

        3. My business isn't reliant on the internet - I can use my computers offline to get work done, which is pretty much impossible if the applications and data is all up in the cloud.

      3. Vic

        Re: There are so many reasons...

        > 3. If your ISP is down, your business is down

        No it isn't.

        Communications is a little harder, but I can do plenty without net access.

        Vic.

    2. itzman

      Re: There are so many reasons...

      ..and data that you couldn't care less if the world and his dog DOES see.

      I.e. 99% of domestic data.

      NO company should let a cloud service anywhere near their corporate data. Especially a BIG company with so many sysadmins you will never know which one sold your data to a third party.

    3. mmeier

      Re: There are so many reasons...

      Privacy can be an issue for companies not to mention legal reasons preventing storage of data in foreign countries. For my privat data l do not care, nothing I keep on a computer is secret or needs privacy so l could as well put it in a cloud

      Data security breach by insiders can be done with local storage just as well

      Reliability depends on what you need or want / need. If you need data access from offsite than cloud is likely more available on the average due to it's distributed nature having no single point of failure unlike storage in a single data center as it is common in private use and smaller companies

  16. Mikel
    Pint

    Azure waves of pain

    Now is a good time to suss out exactly how many Microsoft properties trust their own "enterprise class cloud provider" enough to eat the dogfood.

  17. Anonymous Coward
    Anonymous Coward

    SSL Certs - the new single point of failure

    There must be a name for the law that says "whenever you think you've eliminated all single points of failure, another one arises"?

    1. PhilipN Silver badge

      Re: SSL Certs - the new single point of failure

      Murphy's Law

    2. itzman

      Re: SSL Certs - the new single point of failure

      smiles.

      Like the time the digger on the M1 dug up all the 'diverse' optical fibres that connected the UK south to the UK north and cut the UK internet in half?

      Like the computer in charge of monitoring and switching connections between diverse links failed to connect ANY of them?

      Like the fire that destroyed a company HQ taking all its data with it, and the offsite tapes turned out to be blank..

      1. Jamie Jones Silver badge
        FAIL

        Re: SSL Certs - the new single point of failure

        "Like the fire that destroyed a company HQ taking all its data with it, and the offsite tapes turned out to be blank.."

        I used to work for a company that was affected by the IRA Manchester bomb around 20ish years ago. Equipment was trashed, but the backups were in the firesafe, and were fine..... It just took over 2 weeks before police/forensics/health and safety would let anyone in to get them!

        In the meantime, all the disaster recovery team could do was rebuild the server (as new) and create new accounts for the staff.....

  18. Piro
    Pint

    .. and nothing of value was lost.

    Oh Microsoft. Why would anyone trust Azure anyway? It's not like they already tried storing a bunch of customer's data for Danger.. and lost a bunch of it. Whoops.

    1. cpreston

      Re: .. and nothing of value was lost.

      I think your facts are a bit off there. The danger systems were not running on Azure, or even on updated MS equipment. And it appears that it was an outage that was reported as a data loss story. I have seen no evidence that anyone actually lost data after the restore was completed.

      1. This post has been deleted by a moderator

        1. Anonymous Coward
          Anonymous Coward

          Re: .. and nothing of value was lost.

          The people who support cloud services like Azure are the engineers that develop it - so they have an incentive to prevent it falling over. No one wants a call of this nature when they're the escalation point.

          You can pretty much be assured that they will engineer a way to prevent this happening again - automated SSL certificate expiration checking anyone?

          1. This post has been deleted by a moderator

      2. Piro

        Re: .. and nothing of value was lost.

        I never suggested Danger's data was on Azure, that predates this of course, I'm just suggesting that Microsoft have a proven failure at software as a service. Just ask anyone who's running Office 365!

  19. Will Godfrey Silver badge
    Happy

    Look on the bright side

    At least they weren't holding it wrong.

  20. cpreston

    This doesn't kill "the cloud" for me

    After commenting on a few other comments, I thought I'd make my own. First, I don't work for MS or any cloud vendor, so spare me the ad hominem attacks.

    What I am is a huge user of cloud technology. In fact, my entire company runs in the cloud. Our phone system, our file sharing system, our CRM system, our payroll and accounting system, etc are all cloud providers.

    Am I without outages? Absolutely not. Am I without problems? Nope. Have I fired some cloud vendors over the last few years? Yep.

    But...

    Do I have to hire and manage an IT staff? Nope. When I found out I didn't like a product, was I stuck with it because I paid for it up front and have to wait for it to depreciate? Nope. I just fired that cloud vendor and got another. Do I have to do any planning when I expand and contract the services I use? Nope.

    I am far happier with my company's IT services than I was when I ran my last company and we did it all ourselves. And when there's a problem? I just call the guy. (2 1/2 men reference if that wasn't obvious.)

    This is possibly the dumbest reason a cloud vendor has ever had an outage. But I have to quote the Big Man, "Let you who is without sin cast the first stone." I love all the comments that say they'll never trust the cloud because of things like this. As if you'd never have an outage if you did your own IT. Riiiiiigggghhht.

    1. (AMPC) Anonymous and mostly paranoid coward
      Gimp

      Re: This doesn't kill "the cloud" for me

      Good points cpreston,

      Microsoft Azure is not the only cloud service available. And I have never seen or heard of a data center that has met a 100 % SLA. Not in my past 25 years in the business. We will have to wait for Skynet Singularity and the deletion of all humans before that happens.

      All these big cloud players must have at least one MAJOR fail on their books (look at AWS in Northern Virginia, FFS) as do .... *cough" .. most, if not all ,privately managed data centers.

      Some of these massive failures make the news, and some ... don't.

      Statistically, they are most often caused by human error.

      Afterwards, failure can be measured economically (the right way) or by the degree of exposure and ensuing media frenzy (the most common way).

      I will admit this is a rather SPECTACULAR fail because of the company, number of data centers, customers and zones involved. The apparent simplicity of the reason for which it happened just makes it juicier (but when is there ever a good reason?).

      "Too big to fail" might work for banks and investment houses. But that 21st century oxymoron will never apply to data centers, networks, nuclear power plants, ATC systems, etc. Some things just can't be covered up. Hidden or ignored flaws have a way of finding you, usually late at night , when you are sleeping, on vacation or having a coffee break.

      Whilst some of us can still rush in during that bank holiday weekend and bring the payroll system back to life before Monday morning (after a long-ignored, "too expensive to fix", SPOF finally pukes all over the system) and leave quietly with no one the wiser, that doesn't make us particularly smarter, it just makes us lucky.

      Fortune 500 managed systems proposing 24/7/365 availability are obviously held to a higher standard. I suspect some highly-placed corporate dweeb-bonehead-ms-drone-bean-counter (choose all that apply) decided that automatic cert renewal might be too *expensive*. Which could also explain the magnitude of the policy error.

      School boy mistake indeed.

      Personally, I would call it a hanging offense, at the very least worthy of a hefty and vigorous b*tch slapping followed by 4 year re-assignment to an Alaskan data center. Pity the poor fool who ends up taking the rap.

      Full disclosure: yes I did see this happen (once) in our shop, to a little-used web service.

      We were lucky and smarter afterwards. Policy put in place, etc. forgiven and forgotten, bla

      IMHO cloud computing will become a much healthier and more viable eco-system when it is no longer dominated by a handful of big players. Accidents like this will improve cloud deployments in the long run, particularly if companies start hiring and listening to talented people again.

      Remember there is more than one way to do cloud, here are just a few:

      1) Go public all the way, preferably with non-core services first until you gain confidence. There are many options out there, just go look at them.

      2) Set up your own private cloud first and think about what you can peel off to a public cloud, when and if it becomes economically justifiable

      3) Some hybrid of the two.

      Of course, step 0 is to conclusively prove that any proposed cloud option(s) are less expensive and troublesome to run than by using your conventional data center. That is not always easy to do. But as soon as you have that proof, ît is time to to consider cloud services and solutions in the same way you consider any IT purchase. Of course, some research, assembly and homework will be required.

      Remember that as long as it's cheaper and easier to do cloud options, companies will.

      Although some companies may be CAREFULLY re-thinking that Azure subscription (or at least re-negotiating it).

      Signed,

      A self -confessed, mildly obsessed, fanboi, cloud-tech lover

      1. Anonymous Coward
        Anonymous Coward

        Re: This doesn't kill "the cloud" for me

        @AMPC

        As someone who gets to pick the pieces up after such events as this, I can't help but disagree with all that you and cpreston have presented here.

        I think cpreston summed up what cloud is, when he stated he didn't have to employ IT staff anymore. It's a non technical managers wet dream.

        Now they get to sit in their plush offices doing less actual work (not managing them annoying technical people for example), pointing out to those who own the business how much money they are saving the company (by not paying for IT Staff, or equipment for example) whilst down playing the risks they are exposing the company to.

        I think such people are deplorable, that they would be willing to gamble in some cases the very existence of a company, and all of the staffs jobs, so that they can have an easier life (not having those annoying technical people to deal with for example, no evaluations to do, no wages to pay, etc. etc)

        The only possible way for any company to be able to ensure that data is secure, and available is to do it for themselves. That costs money, but it provides security for the existence of the company, and for the staff who rely upon the continued existence of the company for their jobs.

        Anyone who is dumb enough to outsource the existence of their companies viability to some third party, deserves to fail. Any manager doing this for your company should be questioned very seriously about why they would want to expose your company to such risks.

        1. cpreston

          Re: This doesn't kill "the cloud" for me

          @Dave Dowell

          I'm sorry you find me deplorable. I'm also sorry that you misunderstood my comment about hiring IT people. I am IT people with 20 years of experience in IT. I also now employ another IT person. But we choose to use our IT skills in different ways other than supporting the day to day IT operations of our small company (<10 employees). We have "outsourced" that to various other companies. I'm sorry if you think that means that I deserve to fail. I disagree.

          My choices were a lot like other small companies:

          1. Hire an IT generalist that will know a little bit about everything

          2. Hire a bunch of specialists via cloud companies who know a lot about a few things

          I chose #2, and I don't see how that makes me an idiot. I love how you think that people who are employed by your company are inherently less risky than cloud vendors paid by your company. It's a bias that I'm trying to expose.

          Please tell me why having someone on my payroll makes them less likely to screw up than someone who works for a vendor that does nothing but the thing I hired them to do (e.g. run mail services for me).

          A previous commenter mentioned about the lack of control you have in a cloud situation. That is true and it's not true. It's true in that you are one customer of thousands or millions, and you have little control over how they do IT. It's not true in that if you don't like how one cloud vendor does things, there are lots of other cloud vendors to take their place.

          1. Anonymous Coward
            Anonymous Coward

            Re: This doesn't kill "the cloud" for me

            @ cpreston

            I'll let you in on a secret, the managers in the company you've out sourced your compute function to, do exactly what you do. They look around the market place and employ the cheapest resource they can get away with. Sure they could pay vast sums to employ proper specialists, but like you they have targets, and budgets, and so want their bottom line to look good, so that they can get their big bonus.

            All them people you don't want to employ because you don't want to pay for their skill are just like the people running clouds. Except they don't know or care about your individual infrastructure. Your individual infrastructure is just one of many they run, it's no more personal to them than any of the other infrastructures they run. When it's in a heap on the floor, they're not completely dedicated to getting it back up, they're still making sure they deliver on the other infrastructures they run. They have targets to meet as well, and delivering on all them other infrastructures, even when yours in in a heap, is part of their targets.

            Cloud has a place, indeed it can be a very useful resource, but any place you put it, which could result in your company not being able to survive in the event of a failure, isn't a place where cloud belongs. From your earlier statement about how you've put your entire company IT into cloud, that's what you've done. You've delegated the responsibility for your companies survival to some third party supplier. Who as I pointed out earlier is employing exactly the people you don't want to employ, but putting them under much more pressure to maintain lots of different infrastructures. They know less about your companies specific requirements, they know less about your companies specific needs, and they care less about both than anyone you employ who would be dedicated to understanding and caring about those things.

            Still it's cheap, and you don't have to worry about planning, implementation and running costs. So who cares. The staff at your cloud providers certainly don't, they'll keep their jobs no matter how badly your company fails, they have other infrastructures to run after all, so their jobs are safe.

        2. (AMPC) Anonymous and mostly paranoid coward
          Gimp

          Blue Pill? Red Pill?

          Except that there is so much more to the "cloud" than Microsoft Azure.

          First of all, I do not like using the term "cloud". It is basically a marketing buzz word and means whatever people want it to mean. Unfortunately, we need some kind of shorthand term so cloud it is, at least for today.

          Second of all

          Cloud != the end of IT as we know it.

          Arguing that this Azure meltdown invalidates all cloud solutions is like saying the Toyota brake system recall invalidates the internal combustion engine.

          Azure is a public "cloud" service that offers storage and compute resources for rent. For many companies, that can be incredibly useful and cost effective. Microsoft is one of multiple companies offering the same service. It is using technology that is readily available to private IT shops. This market is growing by 40 % annually.

          Although Azure just failed at providing that service (for about 12 hours based on recent reports), it doesn't mean cloud services and technology can't be used successfully by people who know what their doing.

          With the proper staff and know-how you can build your own Azure cloud and still save money. You may even do it better than MS (or not).

          Like all tech, when used properly cloud solutions can save money and even generate new business.

          But a LOT of people just can't or won't grok this.

          I suspect that weavers and carriage drivers also refused to accept steam engines and cloth looms, but hey, whatcha gonna do?

          Refusing to understand mega trends is a very counter-intuitive approach in a market where there is a massive shortage of people with cloud skills but clearly no shortage in ostrich techies.

          A business needs IT systems that help them make and save money . This fundamental rule still applies to even the most "special" environment, particularly during a bust economy

          You also imply that managers choosing cloud options have never thought about the risks. But I fail to see how managers can even assess these risks without some knowledge.

          When was the last time a manager asked you about the risks associated with this or that redundant power supply, server clustering technology or the company's off-site backup strategy ?

          Would "never", "once" or "hardly ever" be right answers?

          Imagine if these same managers then asked their IT people about the cloud and were effectively told it "it is too risky and it sucks". Then next weekend, one of their country-club drinking buddies explained how cloud services saved their company 1,000,000 USD on last year's IT budget.

          I suggest you at least learn about cloud options so you can explain them to PHBs, otherwise someone else will.

          It really is that simple.

          1. Anonymous Coward
            Anonymous Coward

            Re: Blue Pill? Red Pill?

            @ AMPC

            Why is it cloud fanbois assume that you know nothing about cloud if you speak out about it being used badly?

            You should avoid patronising people who express views you disagree with by implying they havn't got a clue what they're talking about, or sooner or later you might make the mistake of patronising someone who works in a cloud company.

            Thank you for your advice, I am certain however that I understand cloud, so I shan't be rushing off to learn about it.

          2. Vic

            Re: Blue Pill? Red Pill?

            Imagine if these same managers then asked their IT people about the cloud and were effectively told it "it is too risky and it sucks". Then next weekend, one of their country-club drinking buddies explained how cloud services saved their company 1,000,000 USD on last year's IT budget.

            That's very easy. You just point out how much more cash they would have if they didn't bother to pay any insurance premiums for the year.

            Cloud has its part to play in the future of IT, but I've yet to see an instance where the purported savings didn't come at the expense of increased exposure to risk.

            Vic.

    2. This post has been deleted by a moderator

      1. mmeier

        Re: This doesn't kill "the cloud" for me

        Interesting. cpreston did not refer to Azure in his post but to having used more than one vendor in the past. Some stuff he mentioned is not even available in Azure...

        As for reliability: Not even the control units at a nuclear plant are rated at 100 percent and no engineer that knows his job will give you that. 99.999for highly redundant specialist units [2 sets of triples with voting] is the best I have seen, two nines are more likely

        1. This post has been deleted by a moderator

          1. mmeier

            Re: This doesn't kill "the cloud" for me

            Where have I been"forgiving"? By stating the fact that no engineer would guarantee 100 percent? Like cpreston l never mentioned MS nor could l use it on the job for legal reasons [same for Amazon, Google...] And actually the security is not endangered since no one can access the data :)

            For private use it is currently Google App Engine since it supports Java and Dropbox since I got some space for free. In either case 99.9 would be more than enough [8h/year] andl could live with 99 [4 days/year]

            As for engineer: Master in CompSci [Dipl. lnf.] of the TU Braunschweig and a quarter century experience with more OS and languages than the average Linux freak has used distributions

          2. cpreston

            Re: This doesn't kill "the cloud" for me

            It's not a meaningless argument when the point that is being discussed is that people are saying "I'll never trust 'the cloud' because of this MS screwup."

            If your selection of a service or an IT department is that they never have an outage -- for whatever reason -- then you're going to have a hard time finding either.

          3. RonWheeler
            Stop

            Re: This doesn't kill "the cloud" for me

            You're chucking about that word security again in the casual way IT people do to scare idiot managers. The problem quite possibly happened BECAUSE of security. i.e. only a certain level of admin could renew the certificate. That certain level of admin is also quite possibly not prone to doing boring humdrum day to day admin jobs like renewing certs.

            1. Vic

              Re: This doesn't kill "the cloud" for me

              > i.e. only a certain level of admin could renew the certificate.

              Possibly, but *anyone* could *review* the certificate. Indeed, it ought to have been an automated task (I can do it for my customers - why can't a big cloud provider?)

              > That certain level of admin is also quite possibly not prone to doing boring humdrum

              > day to day admin jobs like renewing certs

              If that is so, then there is a *serious* problem in the design of that infrastructure; if the man whose job it is to ensure such a core part of the service considers that job too "humdrum" for him to bother doing it - then he's in the wrong job...

              Vic.

        2. cpreston

          Re: This doesn't kill "the cloud" for me

          Because I wasn't replying to the story as much as I was to the "I'll never trust the cloud" comments.

      2. Anonymous Coward
        Anonymous Coward

        Re: This doesn't kill "the cloud" for me

        @Eadon - A massive MS failure, which even MS fans are saying is a total screw up and somehow you still manage to be obnoxious enough to get a ton of downvotes. My hat is off to you.

        1. This post has been deleted by a moderator

          1. Anonymous Coward
            Anonymous Coward

            Re: This doesn't kill "the cloud" for me

            Here to tell the truth? And make "i shagged your mum" claims? As further up this thread...

            1. Spoddyhalfwit

              Re: This doesn't kill "the cloud" for me

              Yeah, and let's face it, Eadon's posts say "virgin" all over them

          2. Anonymous Coward
            Anonymous Coward

            Re: This doesn't kill "the cloud" for me

            Oh yeah, you can't handle the truth... until your biz goes down.

      3. cpreston

        Re: This doesn't kill "the cloud" for me

        I am doing nothing of the kind. I think what MS did then and before we spectalurly stupid. What I'm responding to is the litany of posts saying "I'll never trust THE CLOUD." MS is not the cloud, nor do MS's screwups ruin the cloud.

    3. Don Casey
      Alert

      Re: This doesn't kill "the cloud" for me

      Cloud vendors are interested in your business (the money you pay them), not your Business (what you do to have people pay you money).

      They don't know your wants/needs/tolerance for outages, etc. etc. If commodity is all you need from IT, then fine.

      If IT is mission critical to you (and that is an ever-expanding universe) then you have to think twice about giving up even indirect control over what happens to the infrastructure running your Business.

      Outsourcing is one thing; you have a direct line of communication to people who should understand your Business and can react accordingly when making changes and when things go sideways.

      Going to The Cloud completely removes you from the decision makers and actors, you have no control over the how and when and all you can do is hope your business is enough to keep them competent enough so they don't screw up your Business.

    4. BinkyTheHorse

      Re: This doesn't kill "the cloud" for me

      Ah, so your ass *is* potentially on the line - I was wondering about that, reading yet another of your comments defending cloud services. You're not a cloud vendor, but you seem to imply that you're responsible for making technology decisions in your company, so if your current vendor has a similar titsup, someone can use this example as "you should have seen it coming" ammo against you. In any case, thanks for the disclosure.

      The backlash against cloud services you see in the comments is really due to one simple promise: that switching to a cloud provider would effectively rid your IT infrastructure of single points of failure, modulo maybe a couple of minutes max downtime in the event of a very serious emergency. Problem, this is another cloud outage in the last couple of months with exactly the opposite characteristic.

      In summary: I concede that cloud services are a Very Useful Thing, but the fact is they *have* been massively hyped and overadvertised. The current reaction is simply yet another IT silver bullet having a new one ripped.

      1. mmeier

        Re: This doesn't kill "the cloud" for me

        Yes, cloud service has been overhyped, even more when it comes to small/medium companies and cloud. You exchange one Single Point of Failure(SPF) like your server for another (your internet connection). You also have to trust your choosen cloud provider that he can deliver availability as contracted. BUT:

        Data centers in the banks typically sell 99.99 (around 1h/year) or 99.9 (9h/year) to internal customers for most systems(1) so that should be enough for most companies even if you may be more affected by planned downtimes(2), And to get the 99.99 figure they use two centers on different locations, double data lines and all the stuff.

        Outsourcing to reduce management jobs does not work. You exchange local IT where you can do the "800pf gorilla"(3) with remote IT where you threaten with contract clauses. Language differences get added even for those who speak the queens english as their primary language(4). If you do, you do it for money and likely for a non-IT business.

        If not for the lousy internet speeds (and german laws) I could see the craftsman in my hometown doing it. Small/medium companies in germany use "cloud" already for tax and salaries with Datev (not called a cloud but technically the same) so doing it for bills and general writing would not be a new concept. And those companies could live with a 99 percent reliability since they have no time critical stuff

        (1) Back in the early 2000 even stock trading was in that range

        (2) That time from internal IT included planned downtime that where put "late" and not affect business

        (3) A 190+ cm, still muscular if "manly girth" equiped man leaning on ITs desk does motivate many admins...

        (4) Most Indian IT workes speak something that resembles english as much as Texan does (Or Sächsisch resembels German)

  21. Cipher
    FAIL

    This what happens...

    ...When you stray from your core business seeking a new revenue stream and you don't, apparently, know jack about what you're doing.

  22. sysconfig

    Everybody sees how M$ f***ed up yet again and thinks "the cloud is not an option". It's not *the* cloud. It's M$ who fucked up yet again in short succession, in their so-called cloud, which seems to be full of schoolboy errors and single points of failure.

    The biggest SPOF sits at the helm of that company, still, for a reason which is beyond many people.

    1. Vic

      > It's not *the* cloud. It's M$ who fucked up yet again

      Sure.

      But the issue is about who cares..

      If I screw up on customer site, the man infront of me cares. And that means I care.

      If a cloud provider (possibly on a different continent) screws up, he's not nearly as motivated as I would be because he is remote from the problem. And the only people the end customer can talk to is a tech supoprt department who will know very little, have no ability to effect any fix, and probably won't even know who is trying to fix it.

      Outsourcing your work to any remote provider means you will have less influence over them than you would if the job were done locally. Whether that makes a difference to the business case is something that needs to be managed and documented.

      Vic.

  23. Wensleydale Cheese

    All your eggs in one basket isn't a good idea

    A real world demonstration that having all your eggs in one (third party) basket is asking for trouble.

  24. Disgruntled of TW
    Alert

    O365 - backed by a financial guarantee ...

    ... yeah RIGHT. Microsoft promise to pay you the paltry amount you paid them for each user. The atrocious effect on your business is not their problem and never will be.

    If you choose cloud, choose very, very carefully. Putting stuff there without understanding the risk to your business could be a career limiting move.

  25. plrndl
    Linux

    Ha ha ha. Ha ha ha ha ha. etc

    This is the best laugh I've had in years. Made my day.

    1. Anonymous Coward
      Anonymous Coward

      @plrndl

      Penguin icon, schadenfreude and childish repetition of ha. Not the most original combination, is it? Is something with MS's name on it going wrong really so great for you? I can't even say I feel sorry for you.

      1. This post has been deleted by a moderator

  26. andersonshelter
    Mushroom

    Data Utilities

    Will it take a catastrophic failure of a Cloud service to make people (and government) demand some sort of regulation and accountability like other "public" utilities? The Internet was designed to cope with nuclear attack - doesn't feel like it today.

    Jus' sayin'.

    1. mmeier

      Re: Data Utilities

      TCP/IP is designed with loosing parts of thenetwork infrastructure. This is an application layer problem.

      1. This post has been deleted by a moderator

        1. mmeier

          Re: Data Utilities

          You will learn to understand the answer when you outgrow PFY status in a decade

    2. Cipher
      FAIL

      Re: Data Utilities

      Yeah, more Government intervention into the net, that's the ticket. It'll be freakin' perfect after the bureaucrats mangle it, censor it and tax it to death.

      Remember when Clinton wanted copies of our encryption keys? Obama is writing Executive Orders right now to give him a "Internet Kill Switch" in the name of national security.

      I don't know about where you live, but here in the USA, anything the Government touches turns into crap. Expensive, censored, politically correct crap...

      1. Anonymous Coward
        Anonymous Coward

        Re: Data Utilities

        The day that happens is when a massive commercial advantage is handed over to countries that don't have a kill switch.

        One day, some politicians will get the point, so I suggest you write your elected representatives and help them understand. Vote out the ones who don't.

        ,.

        Otherwise prepare for Armageddon, at least on a political scale

  27. Zc456
    Holmes

    So, just another day at the office...

    And the world remains unsurprised.

  28. JDX Gold badge

    Dearie me

    Out of interest, have the other big players had similar screw-ups? I know things like gmail have gone down but has the entire google app cloud or EC2 flopped like this?

    Maybe it's deliberate ploy by MS to make cloudy computing generally look bad so everyone stays with desktop computers, running Windows :)

  29. Levente Szileszky
    FAIL

    Once again I see my anti-MS-cloud stance completely justified...

    ...as I NEVER trusted this giant Ballmerian spider mess of services, not for a sec - remember the (fittingly named) B(ig)PoS? They had more downtime in one year, and always worldwide than everybody other hosted service provider (Google, Rackspace etc) TOGETHER, UNTIL DATE (years and years)...

    ...'oooh, Azure/WebOffice/O365/whatever-its-latest-newest-old-product is totally different, it's top-notch now' I heard every single time - and they just KEEP FAILIN' KEEP FAILIN'...

    ...so while I'm far from happy with Google's weird support I'm still enjoying my CEO's trust because I went with Google Apps Business and never had to deal with HOURS-LONG outages affecting our most critical communication//cloud services, thanks to Google.

  30. FanniM

    Cloud computing is just not secure enough yet.

  31. Anonymous Coward
    Anonymous Coward

    nothing new here

    Anyone remember half the internet going down in April 2011 when EC2 went down for 7 hours? or the Netflix outage on Christmas eve because of AWS problems? Gmail in December 2012? Google Apps in October 2012? AWS in October 2012?

    100% uptime doesn't exist. Shit happens.

    1. Levente Szileszky
      WTF?

      Re: nothing new here

      As far as Google goes they *never had* any world- and system-wide *full* outage, at least we've never experienced anything like that (we're in NYC) during the past ~3-4 years, since we switched to GA Business.

  32. Mr Young
    Thumb Up

    @Eadon

    You forgot to set your Interrupt Enable Flags before the ';loop forever' or 'goto 10' stuff. The caps are inspired by amanfrommars by the way - obviously not my own work!

  33. Winkypop Silver badge
    Megaphone

    I dedicate this thread to Eadon

    He/she must be like a pig-in-poo right now.

  34. amanfromMars 1 Silver badge

    Do you Know what Tomorrow will Bring? Have you prepared Anything and/or Everything for Presentation

    @AC 17:54 - I'm not here to win a BEAUTY contest, I am here to tell the TRUTH! And THEY CANNOT HANDLE THE TRUTH …. Eadon Posted Saturday 23rd February 2013 18:08 GMT

    There is an added cloudy ingredient in the future azure blue skies thinking mix which is intelligently designed to render confusion and CHAOS to wannabe controllers and Dummy Operating Systems, Eadon, and that is that they don't know what the truth is to be.

This topic is closed for new posts.

Biting the hand that feeds IT © 1998–2019