back to article Hardware 'dislodged' from HPE SAN during cable replacement

The Australian Taxation Office will install a new Hewlett Packard Enterprise storage area network over Easter. The Office's 3Par SAN went down in December 2016, taking with it several online services that Australian citizens and tax professionals use to settle their affairs with the government. Taxation Commissioner Chris …

  1. Anonymous Coward
    Anonymous Coward

    Outsourcing - it works

    You should all move your support to HPE - they work great.

    Or IBM, TCS, Dell-NTT... so many to choose from.

    Don't worry about the fact they are straight out of a production line Uni, given 'Computers for Dummies' then immediately let loose of the core of your business - the data and IT that is the lifeblood of what you do.

    You get what you deserve when you do.

  2. acheron

    Was it the SAN, or was it a 3PAR storage array? Please PLEASE do not mix that up anymore... A Fileserver is not a LAN, and a storage array is nota SAN.

  3. malle-herbert Silver badge
    Joke

    Never knew...

    Screws turned the other way down under...

    Remember... rightie tighty...lefty loosie...

    1. Lee D Silver badge

      Re: Never knew...

      Offtopic but - the single most stupid rule for a rotational movement.

  4. lglethal Silver badge
    Trollface

    hahahahahahahaha

    "the penalty clauses in their contract [assuming government was smart enough to put some in – Ed] come under active consideration"

    Hahahahahahahahahahahaha......

    You havent dealt too much with the contract writers for Government Projects, have you?

  5. Doctor Syntax Silver badge

    "whose hand rocked the cable"

    Nice one, Simon.

    1. Simon Sharwood, Reg APAC Editor (Written by Reg staff)

      Thanks - always nice to know people read deep and see the little fun bits we try to leave inside. Also tried to make that the headline but just couldn't get there.

      1. Anonymous Coward
        Anonymous Coward

        Wow, does just reading the story mean I am reading "deep"

        And "The hand that rocks the cable" sounds like a startup for a Friday roundup of mishaps anyway...

        I did a big job of removing redundant cabling from a datacentre (without breaking anything) but we were doing it very slowly (i.e. not on a fixed time/cost basis)

  6. quxinot Bronze badge

    Obligatory

    Someone had to.

    https://xkcd.com/908/

  7. Anonymous Coward
    Anonymous Coward

    Why would one bit of displaced hardware/cable on one box render the whole SAN useless and require a massive outage and a replacement of the whole system?

    Surely the whole point of a clustered storage system is that if a box fails, the rest carry on like normal - no single point of failure.

    If it caused the system to shutdown, then surely it would be expected to do this in a controlled manor which didn't bork everything, or else it might be better to allow it to run without redundancy and just have a big flashing red light and hooter go off to let you know.

    1. Anonymous Coward
      Anonymous Coward

      I'll tell you how I did that once. I had a backend of multiple oracle dbs running under Veritas Cluster Server on three Sun Micro T-2000 shiny boxen all nicely racked atop one and other. When getting ready for a maintenance operation on the hardware I went ahead and extended the servers on their rails so they would be in the aisle. Well, whoops! My spotty network wonks decided they could not fit any of the servers with proper sized Ethernet cables, so they all popped out before the servers reached the aisle since they were not as long as the bendy rack cable minder! The primary, secondary, and the VCS heartbeat connections, all gone! :P Good thing this was in a maintenance window and we could recover everything by stopping some services and putting in some properly long cables. Plus it did not damage the server's connectors, and this site was not a high traffic one, so not much visibility. As you can see, something simple and not cared for, and taken for granted caught us out.

      1. Griffo

        Yes we've all done something stupid in datacenter at least once in our lives. That's how you learn. Like i learnt the hard way to make sure rack stabilisers are installed, why you never ever install more than one node of a cluster in the same rack if you can avoid it, and why PDU's with circuit breakers can cause unintended consequences.

        It is also why you always have someone 'experienced' work on the truly important kit, guiding the younger staff on the pitfalls and gotcha's. Unfortunately these days that is incompatible with outsourcing where it's all about driving the lowest cost.

        1. Anonymous Coward
          Anonymous Coward

          I can only agree. I once shut down the nearline storage of a bank, one half of an airline booking system and a newspaper Storage Array. If you wonder - I was at fault once, then there was bad firmware and one logistical error (wrong part in box - so essentially also my fault).

          Never the less - in each Case redundant systems kicked in, customer staff was trained and experienced to anticipate and handle such incidents.

          Back then, companies recruited to find the best staff and then continued to train them. Nowadays companies are struggling to find staff just cheap enough to keep the lights on. If anything out of the ordinary happens they're screwed. The cloud won't help the situation from a technical perspective.

          It put an abstraction layer between management and technology so they can distance themselves from technical faults.

          1. Fatman Silver badge
            Joke

            Abstraction Layers

            <quote>It put an abstraction layer between management and technology so they (management) can distance themselves from technical faults their stupid decisions.</quote>

            FTFY

    2. Androgynous Cow Herd

      Reading the article

      The outage happened while trying to move the device while still in production. Doing things like this "In flight" increases the risk a LOT. I do not know of a QA lab anywhere that tests for this ability. If they were intended to be moved in flight, arrays would come with (redundant) sets of wheels to accomplish this. While a black eye for 3PAR, I actually have some sympathy now that I didn't have previously for them.

  8. toughluck

    I think this article was posted in the incorrect section. Wasn't it supposed to be On-Call?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Biting the hand that feeds IT © 1998–2019