back to article Visual Studio Online goes titsup as Microsoft wrestles with database

Microsoft's Visual Studio Online services for software developers are in the midst of a total outage that has lasted for more than four hours. The Redmond giant has blamed a database snafu. The services, which were launched in November 2013 to coincide with general availability of Visual Studio 2013, are hosted on Redmond's …

  1. Bob Vistakin
    Linux

    Tut tut tut

    Tuttity fucking tut.

  2. Mephistro Silver badge

    Another day...

    ... another cloud SNAFU.

    1. Steven Raith

      Re: Another day...

      "Another MS cloud SNAFU" more like.

      I'm no cloud proponent, but core Azure based systems seem to shit themselves, comprehensively, on a monthly basis.

      In short, if you really, really, want something cloudy, don't touch MS with a fucking barge pole, period.

      Steven R

      1. Steven Raith

        Re: Another day...

        Fucking triple post.

        I must be proxying via Azure or something.

    2. This post has been deleted by its author

    3. This post has been deleted by its author

  3. Khaptain Silver badge

    Cloud

    Just the damned word makes me cringe now.

  4. Anonymous Coward
    Anonymous Coward

    Sourcecode lost?

    That's the word on the street.

  5. Tom Chiverton 1

    According to that status page, auto-scaling has bee broken for two weeks now as well.

  6. Yet Another Anonymous coward Silver badge

    Opportunity

    I heard of a startup near Seattle that is selling something called a microcomputer which will be small enough that you can have an actual computer in your own office

  7. The_Idiot

    And as the march to subscription (online) services goes on...

    ... I watch it march away into the distance, as I continue to stand as far away from the bloody things as I reasonably can.

    I understand the value of such services to the service provider. I quite fail to understand their value to the consumer, at least, to this Idiot consumer.

    1. Anonymous Coward
      Anonymous Coward

      Re: And as the march to subscription (online) services goes on...

      The value to the consumer is that we can put together consumable services for very little cost.

      I provide mobile lottery services to around 100K mobiles. it costs me about £40 a month. I just couldn't do it for that cost otherwise. If I want to scale out I can do it almost instantly. If I want to scale up I can do it almost instantly.

      I'm about to release a new app using Azure and it should cost me about £20 a month to run initially. If it takes off I can use Azure to scale up for very little cost.

      It helps that the cost of cloud computing only seems to go down as Amazon, Google and MS continue their price war.

      1. BlueGreen

        Re: And as the march to subscription (online) services goes on...

        With respect, I'm a bit puzzled.

        > 100K mobiles. it costs me about £40 a month

        It doesn't sound like you're exactly stressing out any system if you're getting it that cheaply. How do you know what it would cost on a small dedicated server?

        > If it takes off I can use Azure to scale up for very little cost.

        And if it breaks, what then? (it's possible that with your app a day's delay isn't a problem)

        > It helps that the cost of cloud computing only seems to go down as Amazon, Google and MS continue their price war.

        It's called a race to the bottom. From copious experience, buy cheap pay twice (shrug).

        1. dan1980

          Re: And as the march to subscription (online) services goes on...

          "And if it breaks, what then?"

          Exactly.

          Now, that's not an argument against using a cloud service but a question you have to ask yourself (and seriously assess) before throwing your eggs in that particular basket.

          It's quite possible, as @BlueGreen says, that the answer to that question for you is "not much", in which case the cloud is just fine for you.

          The problem, however, is with people either not understanding the risks of cloud services or not seriously addressing them. Simply saying "I can do it cheaper on a cloud service" can be misleading because there are subtle but important differences in the 'it' you are doing.

          As-A-Service offerings are useful but you have to understand the key word: 'SERVICE'. That means that you should have a service level agreement and you must understand what it says.

          Azure (standard tier) has a 99.9% uptime - each month. That means up to ~45 mins of downtime are possible before the SLA is breached. The big caveats are:

          • That 45 minutes can be at the most crucial time for your business.
          • The recourse for breach of SLA is a discount on your bill, which may mean much, much, MUCH less financial compensation than lost revenue.
          • The 99.9% SLA is per-service.

          That last one is important because it means that if you have, say, 3 services that all need to inter-operate to deliver your 'application' - e.g. a DB, a processing and a web server - then your potential downtime is TRIPLED, because each can be out for 45 mins and the loss of any one will take your application offline.

          And so on.

          You can increase the redundancy and therefore uptime, of course, by provisioning multiple VMs for the various services and make them geographically-distributed but even then, if you are working within a single provider then you run the risk of a service-wide outage. To mitigate against that, you can design your system to use multiple providers, but each of these steps increases the cost far beyond the 'I can host my application for £40/month'.

          The point is that you have to know what you are getting and what it means for your business and anyone who either promotes or accepts the "I can do it for $X' argument" needs to be cautioned to take a deeper look.

        2. roselan

          Re: And as the march to subscription (online) services goes on...

          That's the point. If it is a major success, he can still move in house. Most do. And if no one comes, you don't have to sell/deprecate the infrastructure.

          He doesn't needs a dba, a sys admin, a net admin, a security admin, an admin admin (manager?), listen to channels offers, and get surprises.

          The best advantage of clouds? you don't forget to change tape in your prehistoric backup thingy. And if it breaks, it can actually start again.

          The main thing to keep in mind is that you make your thing portable (data included), so that you can move in with not that much pain when needed.

        3. asdf Silver badge

          Re: And as the march to subscription (online) services goes on...

          @BlueGreen very well put

        4. stephajn

          Re: And as the march to subscription (online) services goes on...

          If this is the case, then perhaps you might not want to use the power company's power either. After all, if they go titsup, then you're out of power. Same for gas...

          When it works....people love it and it is great and wonderful blah blah blah. Successful businesses like Netflix use cloud to run their service. (Amazon if I am not mistaken)

          But just like any other service that someone else provides, there can be problems. Power Grid fails....and the lights in your house go out. Some upstream provider that your ISP depends on goes ka-boom and your internet is down. (Happened nationwide just a couple of days ago with Shaw here in Canada. A routing problem made it impossible to access a LOT of websites) But that doesn't mean we don't use these services.

          If we were to withdraw from consuming a service just because it happened to fail every once in a while, then we wouldn't be using the power grid....or cable television....or cell phones....or {insert service here]. For that matter, no one would ever have a web hosting company host their site for them....they'd all invest in a server farm to do it themselves!

          For those that rant about how evil cloud services are blah blah blah blah.....then I say to you.....turn off your power and run your own generator if you think doing everything yourself is that much better.

          *climbing down off soap box now*

      2. Anonymous Coward
        Anonymous Coward

        Re: And as the march to subscription (online) services goes on...

        "I provide mobile lottery services to around 100K mobiles".

        WHY? Do you agree with Phineas T Barnum that "there's one born every minute", and you should "never give a sucker an even break"?

        I'd imagine you'd be as happy to come on here and admit to being a slave-owner as to pushing "lottery services".

      3. Salts

        Re: And as the march to subscription (online) services goes on...

        @AC 14 hours ago

        Not sure of why the down votes for a statement of fact that you benefit and point out how others could also benefit from cloud services.

        Without cloud services it is unlikely we would have edX, Coursera and the likes, which IMHO are a great benefit to many people.

        I have major doubts about cloud services, but that they work(most of the time) for a lot of people is certainly not one of them.

    2. Anonymous Coward
      Anonymous Coward

      Re: And as the march to subscription (online) services goes on...

      Cloud has its uses. Development is not one of them.

  8. This post has been deleted by its author

  9. Anonymous Coward
    Anonymous Coward

    Clean compile

    Ship it!

  10. Steve Knox
    Coat

    So how long until...

    Cortana goes titsup...?

    (The dirty mac, please.)

  11. SVV Silver badge

    "After about an hour of investigating the issue"

    Naughty, naughty, no contingency planning or failover or proper acceptance testing again.......

    Quiz question : How soon should you be able to roll back changes to your cloudy service if they break ?

    a) almost immediately, as you have servers on standby to do just that at a moment's notice.

    b) about an hour

    This seems to be happening with MS cloud stuff on an almost daily basis right now, giving ample opportunity for posting the same sort of comments about the increasing evidence of managerial ineptitude that must take the blame for it, but I'm sure their army of downvoters will be onto this post even quicker than their service came back online.

    1. stephajn

      Re: "After about an hour of investigating the issue"

      "I'm sure their army of downvoters will be onto this post even quicker than their service came back online."

      I up voted you. :) And I use Azure! Along with VS Online, and so forth.

      "This seems to be happening with MS cloud stuff on an almost daily basis right now"

      It certainly feels that way. If it isn't Azure, then it is something else like XBox Live or a bad update that causes BSOD failures. Trustworthy computing indeed..... </sarcasm>

  12. chris lively

    I bet they were using LINQ. That stuff is garbage.

  13. Henry Wertz 1 Gold badge

    My guess?

    My guess? Rolled out newer DB software. It corrupted something. On rolling out older DB software, DB still corrupted so it didn't come back up properly. Restored DB from backup and (possibly) brought backup up to date using transaction logs.

    Obviously for reliability for a "cloud" DB, it should have been possible to upgrade a small portion of the DB machines (perhaps just 1 machine at first), and see it fail, without really impacting service since the rest would be running the old DB software.

    1. Anonymous Coward
      Anonymous Coward

      Re: My guess?

      Maybe they should try DB2.

  14. William Boyle

    What Microsoft refuses to understand

    If you are providing critical infrastructure services to the public, you need to plan for failure and redundancy in order to provide 365x24 service. If you don't, you are remiss and should lose your business. I was responsible for the design of an application development framework (in C++) for semiconductor, disc drive, and flat panel display manufacturing systems that had to support very large installations, thousands of users, and process terabytes of data per day, where 1 hour of down time would cost the customer over $10M USD per hour in lost profits - failure was not an option. We had to design the systems to be failure-resiliant with no single point of failure at the network, system, database, or other system components. This is not easy, but it is possible - if your system has a chip, disc drive, or LCD display, then that software most likely built it...

    I left Microsoft shortly after they closed their purchase of Nokia Mobile Phones (where I was working), because they still refuse to understand this. My position was Senior Performance Engineer handling 5000+ servers worldwide, and the software I designed and wrote collected 10 billion data points of performance data per day so we could apply mathematical and engineering algorithms to monitor system behavior and predict when systems, networks, and databases might fail. Unfortunately, most of the people I worked with at Nokia will be looking for new jobs soon... :-(

    In my opinion, this failure of Microsoft is inexcusible. I hate to see what will happen to Azure users in the future.

    1. Anonymous Coward
      Anonymous Coward

      Re: What Microsoft refuses to understand

      "My position was Senior Performance Engineer handling 5000+ servers worldwide, and the software I designed and wrote collected 10 billion data points of performance data per day so we could apply mathematical and engineering algorithms to monitor system behavior and predict when systems, networks, and databases might fail"

      Microsoft already have lots of that sort of stuff up and running. Hence presumably why you were surplus to requirements.

      Good examples here: http://download.microsoft.com/download/4/8/8/4885BBB9-2675-42CB-9CF2-F11B69C3C2FB/energy-smart-buildings-whitepaper-1.pdf

      http://www.globalfoundationservices.com/operational-excellence.aspx

      1. roselan

        Re: What Microsoft refuses to understand

        wow, if even ms fans/turfers reach the depth of MS management, this doesn't bode well for internets wars fun. I mean, are you a nokia fan in training trying to discredit ms fans?

        1. he said he left.

        2. citing PR bs, as well as *accenture* PR bs is like a new record.

        3. Hence presumably why you are very good fit for both companies ;)

    2. dogged
      Meh

      Re: What Microsoft refuses to understand

      > If you are providing critical infrastructure services to the public, you need to plan for failure and redundancy in order to provide 365x24 service

      1. There is no such service.

      2. ALM is nice but it's not critical. Even CI build servers off VSOnline are based in-(your)-house so you don't lose those either unless you fuck up. OH NOEZ, I MAY HAVE TO MERGE 67 CHECKINS AT ONCE WHEN IT COMES BACK UP...so what? And the PM can't see your burndown chart when he fancies a shuffle off the wrist? Wowie zowie, the world is ending.

      Welcome to the internet, where mountains are made out of tiny molehills every day (but not reported if they're Google molehills).

  15. Semaj
    FAIL

    Worrying

    We're very heavily invested in all things Azure so this outage caused us big headaches yesterday.

    It wouldn't be so bad for TFS online to be down if Visual Studio didn't have to timeout (~3 minutes) on EVERY SINGLE FUCKING CHECKOUT...

    But no, the amazing resilient cloud can't be down so no need to do some defensive coding around that.

    It's exactly this kind of crap that is turning me off MS dev (and MS in general tbh) more and more. Which is a shame because I normally sing the praises of both Visual studio and TFS, especially when used together.

  16. Anonymous Coward
    Anonymous Coward

    Clever really...

    ...contriving to show up the unreliability of their cloud services, their database, and their company all at once.

  17. This post has been deleted by its author

  18. justok

    Probably from the most recent MS patch tuesday updates

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019