back to article 'Planned maintenance' CRIPPLES nearly HALF of all Salesforce instances in Europe, US

Botched network maintenance has been blamed for a huge crash at Saleforce’s data centres, taking out customers’ CRM and data services across the US and Europe. Seven of Salesforce’s 17 North American instances and two out of four in EMEA went down. Servers started dropping out just before 2am UTC on 15 November, with outages …

COMMENTS

This topic is closed for new posts.
  1. Longrod_von_Hugendong

    Marketing BS...

    Be mindful of it, SalesFarce are full of it.

  2. Ian Moyse
    Thumb Down

    Salesforce is often touted as the defacto CRM, but big is not always better and certainly not right for all.

    Independent analysis of viable Salesforce alternatives can be found at www.g2crowd.com.where leading CRM's are compared by users in an open format.

    Ian Moyse

    Workbooks

    1. Anonymous Coward
      Anonymous Coward

      Nice site - but only lets you sign in/register with a LinkedIn account - fark that.

    2. Anonymous Coward
      Anonymous Coward

      @ Ian Moyse from Workbooks CRM

      Nice advert!

  3. Neill Mitchell

    Well

    It's not called cloud cuckoo land for nothing.

  4. Anonymous Coward
    Anonymous Coward

    ....No carrier

    1. Destroy All Monsters Silver badge

      CVN-78 is indeed over budget and not ready for prime time.

      1. Trevor_Pott Gold badge

        ...and she's not even named Enterprise!

  5. Destroy All Monsters Silver badge
    Devil

    Nobel Peace Prize for .... Twitter?

    I like it how Twitter is nowadays immediately used as a safety valve to let off steam and bad karma.

    This has probably prevented a few cases of cow-orkers "going postal" and transforming their colleagues and middle management into peppered steak and filleted fish before they can be stopped by the boys in blue.

    1. Trevor_Pott Gold badge

      Re: Nobel Peace Prize for .... Twitter?

      Twitter is just as likely to cause such incidents when a postal worker/writer/politician/policeman/what-have-you is forced to deal with the braying masses.

      The internet is full of piranhas, most of them utterly irrational and without any semblance of clue.

  6. Pascal Monett Silver badge

    Dreamforce

    There will be force, but I have a hunch that there will be little to dream about.

    And yes, crashing your product because of planned network maintenance is a really bad sales point.

    But the cloud marches on, oblivious of its problems.

  7. Joe 35

    Presumably the outrage is because companies that host their own systems never have outages ?

    1. Trevor_Pott Gold badge

      The outrage is because of scale. If one business selling tat to people has the POS system go down they inconvenience a few dozen folks that day, maybe cause a knock-on b2b issue to a couple of other businesses.

      If VISA goes down, the world stops. Now the same is true of salesforce, Amazon and increasingly Microsoft. That's hundreds of millions of consumers inconvenienced and millions of B2B issues created.

      How many single points of failure does your economy need?

      1. Fred Flintstone Gold badge

        How many single points of failure does your economy need?

        The problems with the cloud summarised in one sentence - epic :)

    2. Rampant Spaniel

      or perhaps because 'the cloud' was touted as (and one of it's major selling points infact) invulnerable to outages. I don't think anybody is suggesting that inhouse systems fair much better, but when the marketing coke hounds sell something based strongly upon a strength that is then repeatedly shown to be absolute ballcocks in practice it is only fair to point and giggle.

      The reality is that the concept of cloud computing probably is a lot more resilient, however it is left in the hands of human cloud wranglers who are, as always, a major source of all chaos in the universe. The next obvious step would be a distributed network that controls itself and removes humans from the equation. Perhaps a suitable name would be skynet??

  8. Anonymous Coward
    Anonymous Coward

    The cloud is only vaporware and it never will get any better....

    Seems that no one will ever learn that the so called "Cloud" is nothing but vaporware, ready to evaporate all your data at a moments notice.

    The cloud will enslave those who will trade convenience for security

    1. Destroy All Monsters Silver badge
      FAIL

      Re: The cloud is only vaporware and it never will get any better....

      Fail for inability to recognize security and availability as separate concerns, bringing in security here and completely ignoring economics like a juvenile do-it-yourselfer.

      Meanwhile "The Internet is only vaporware and it never will get any better ... everyone needs a leased line from the incumbent operator, yadda, yadda, herpers derpers."

      1. Anonymous Coward
        Anonymous Coward

        Re: The cloud is only vaporware and it never will get any better....

        Truly secure systems are designed (or should be) so they CANNOT delete your data during routine maintenance. That's not the same thing as "Uptime" or "Availability" which is where the data is temporarily offline but still there. I had understood that data was deleted, not just unavailable.

        Nobody said anything about leased lines etc etc.

        The fact is that your data is not "secure" if one must worry if it can be inadvertently deleted by the very system you trust to keep it safe. Only YOU should be able to delete it.

        1. tet3

          Re: The cloud is only vaporware and it never will get any better....

          What in the article gave you the impression that there was data loss? I don't see anything there that suggests it in the slightest. All the outages reported on in the article leave Salesforce at better than 99.9% uptime.

          1. Anonymous Coward
            Anonymous Coward

            Re: The cloud is only vaporware and it never will get any better....

            Just the data lost because companies were unable to work. Now hopefully people are going to think a little harder about how they recover from a real Salesforce failure. And btw 99.9 - I spit on your availability...

            1. RegGuy1 Silver badge

              Re: The cloud is only vaporware and it never will get any better....

              Wonderful news...

              ... if I have a disaster recovery company. What better advert can you have than making sure you have a second copy?

              You may have the best company with the best people in the world, but put all your data in one basket and someone drops the basket -- you're business is toast. WILL PEOPLE NEVER LEARN?

          2. Anonymous Coward
            Anonymous Coward

            Re: The cloud is only vaporware and it never will get any better....

            "What in the article gave you the impression that there was data loss?"

            Have you not read any other news about "teh cloud" recently?

        2. FutureShock999

          Re: The cloud is only vaporware and it never will get any better....

          I am not an immediate fan of the cloud (that is, use it when it makes sense, but sometimes it doesn't).

          But let me answer you with a bit of history. I remember a HUGE data warehousing project at a large bank in Boston more than a few years ago. While in build of this totally secure, non-cloud system, the lead DBA confused his environments and ended up deleting the entire LIVE database of master dev customer data, rather than a test environment. He is someone that used to work with me, this story is not fiction. Gone. The back-ups were out of synch and didn't reload properly. WEEKS of this data warehouse being down, development staff stopped, hundreds of thousands of dollars lost to development time...and that was a very simple mistake, made by a usually very skilled individual.

          If you don't know more stories like that, they you haven't worked in IT very long. Cloud, in-house, mainframe, SOA, or client-server...these are all just technologies. But the fundamental fuck-ups are usually human in nature, and they will always happen. And I have seen them happen about equally on all of those platforms, even to good staffs.

          1. Anonymous Coward
            Anonymous Coward

            Re: The cloud is only vaporware and it never will get any better....

            Hell I've been there, hit a typo, took out a phone system for 4 hours on Christmas Eve, just me and a security guard left on site...

            Back Up? This was the NT4 days.....the 4 hours to manually rebuild would be far less painful than uses a backup in those days.

      2. Anonymous Coward
        Anonymous Coward

        Re: The cloud is only vaporware and it never will get any better....

        Fail for inability to recognize security and availability as separate concerns

        Keep an open mind. It is entirely possible that the author used the word security in one of its other meanings, namely "freedom from risk".

  9. Peter Simpson 1
    Black Helicopters

    On BOFH Day?

    Coincidence?

  10. Will Godfrey Silver badge

    This seems to be a trend

    Will nobody rid me of these turbulent clouds?

  11. Number6

    Did the silver lining fall out of the cloud?

  12. Anonymous Coward
    Anonymous Coward

    But of course...!

    When I was a technician with DEC Field Service (or Field Circus as some of the more bitter customers called it) we used to say: "It's called 'preventive maintenance' because it prevents [whatever it is] from working'".

  13. codeusirae
    Facepalm

    Salesforce lays off 200 employees ..

    "Salesforce Marketing Cloud announced it is laying off another 200 employees — approximately a third of which are located at the company’s New Brunswick offices — following a round of cuts that last fall that affected more than 100 jobs." link

    1. Destroy All Monsters Silver badge

      Re: Salesforce lays off 200 employees ..

      In marketing, not sysoping?

  14. Anonymous Coward
    Anonymous Coward

    SDN

    They must have tried to implement SDN...

    1. Trevor_Pott Gold badge

      Re: SDN

      If that was the case, I'm sure it was an attempt to implement Cisco's version of it. Everyone else has a business case to make SDN that actually works.

    2. Joe H.
      Alert

      Re: SDN

      Which makes me wonder if they were successful implementing some aspect of SDN, or after hours of hand wringing, they decided to cut it back to the way it was before.

      Either way, the internal shouting will take a few dozen post mortems to die down, pun intended.

  15. asdf

    Hahaha knew I remembered something about their infrastructure

    >In a candid keynote speech at the Ricon West distributed systems conference on Tuesday, Salesforce architect and former Amazon infrastructure brain Pat Helland talked up Salesforce's internal "Keystone" system: ...

    >"The ideal design approach is 'web scale and I want to build it out of shit'." ...

    >"Salesforce has a preference for buying "the shittiest SSDs money can buy," he said"

    http://www.theregister.co.uk/2013/10/29/salesforce_infrastructure_reveal/

    1. Matt Bryant Silver badge
      Boffin

      Re: Hahaha knew I remembered something about their infrastructure

      "....>"The ideal design approach is 'web scale and I want to build it out of shit'." ...

      >"Salesforce has a preference for buying "the shittiest SSDs money can buy," he said"....." Maybe he thought it was cool to describe commodity hardware as "shit", but then it's not uncommon for geeks from old-style high-availability backgrounds, where they usually worked with kit such as mainframes where the only supported option was a 6' serial cable that cost $2000, to describe commodity as "cheap shit". Indeed, there seem to be some masochistic architects that take glee in informing all what clever designers they are that they can use "shit" and make wonderful systems (they hang out on Linux forums mostly). The truth is if you design a truly highly-redundant system then you can indeed build it out of "shit", it's just the MTBF is shorter you're much more likely to be using the failover and recovery functions on a regular basis. Of course, if you screw up the design, then you tend to get those seven hour outages.....

      1. asdf

        Re: Hahaha knew I remembered something about their infrastructure

        Well said and exactly what I meant. The issue is not shit hardware but that Salesforce design seems to also be shit. As you imply good expensive hardware can give more leeway for design faults. Still this architect came off as a real wanker and if you are going to brag like this then if I read an article about your site going tits up for hours in just a few months I will remember. Plus as pointed out in the comments of that previous article this is not the first massive collapse of salesforce in recent memory.

  16. GoatBoy3000

    It's all bloatware anyway. Stuff you don't need. Whats wrong with a DB table somewhere "Customers". Done.

  17. Rampant Spaniel

    I think we are all missing the bigger issue here, the authors use of the word tweeple. Such a disgrace should not go unpunished!

This topic is closed for new posts.

Other stories you might like