back to article Imagine every mistake you can make with a new software rollout...

There is not a sysadmin in the world who has not had to deal with a botched software rollout; for most, it is a coming-of-age experience. But few are likely to have made quite as many mistakes as the people behind the Alameda court system's new software. They have managed a rollout so inept that it should serve as a case study …

  1. Anonymous Coward
    Anonymous Coward

    SO VERY TRUE!

    I work in a court in Northern California (albeit not Alameda). This article tells you why most government IT projects are in Epic Fail mode.

    Court clerks were neither sufficiently consulted nor were they adequately trained on the system...

    There you go, folks. You see, the high-ups know EVERYTHING. The peons know NOTHING. Well, they do ask them to write something up, sometimes. But NEVER NEVER NEVER do they let the actual workers see prototypes or ask them if something will actually DO THE JOB. Nope. They, in their infinite wisdom, simply have the system built according to whatever fancies they have relating to what the working classes do all day, then dump that system on the workers, leaving them to sort out the total omnishambles management has once again created.

    </rant> Okay, sorry for the rant, but been there, done that, got the T-shirt and wear the scars. And it's about to happen to us AGAIN where I am. The last time this happened, we had clerks crying because they couldn't even figure out how to do something that took them 5 minutes on the old system. I'm stocking up on tissues. I hear the ominous rumble of an approaching train wreck.

    Perhaps in some alternative reality, management actually consults the actual people who do all the work. Maybe they show them designs and let them have useful input at all stages of development. But not here. Never gonna happen 'cause PHBs.

    1. Anonymous Coward
      Anonymous Coward

      Re: SO VERY TRUE!

      Sadly, it's just not the public sector. But I sense your pain.

      I know of case where a company spent $240 Million and got bumpkis. Damn near took the company down. They did have focus groups of users however, the contractor still did it their way with a wink, wink of the IT Director's eye since his wife was on the board of the contracting firm. So much money was also wasted on focus groups, hiring of trainers (lots... and lots of trainers) since it was the major system internal to the company. Oh.. and the mainframes would all go away and be replaced by wonderful server farms. Big dollars on hardware.

      If the system had gone live to the whole company instead of just one area, the staff size would have needed to be doubled because of the slowness, latency, and the UI. Instead of a couple of mouse clicks... hot keys (dozens of them), keyboard inputs (dozens more usually of the same data already entered). But hey, one of the selling points was the employees wouldn't need to use mice.. except for anything outside of that system. Being Windows based... no mouse would have been a folly. The project rapidly spiraled out of control with 100's of offshore contractors coming to the company spaces and spending all day calling home to the sub continent and drinking tea since they had not been told what their job was or they needed to do each day/week.

      Needless to say, no one got fired, No one was charged with fiduciary culpability. The IT Director went on to bigger and better things. The mainframes are still there as is the server farm that's mostly been cannibalized for other projects. What is new is basically a screen scraper to format the mainframe data to something that's standardized for all departments at far less money. Oh,... the company is smaller now in the number of employees as this system is faster and needed less redundancy is employees.

      1. Antron Argaiv Silver badge
        Thumb Up

        Re: SO VERY TRUE!

        Needless to say, no one got fired, No one was charged with fiduciary culpability. The IT Director went on to bigger and better things.

        I believe the relevant management term is: "F*ck up, and move up"

    2. Anonymous Coward
      Anonymous Coward

      Re: SO VERY TRUE!

      > "Perhaps in some alternative reality, management actually consults the actual people who do all the work. Maybe they show them designs and let them have useful input at all stages of development."

      Conversely, in other organisations, there is an IT department which involves the end-users in system specification, design, and implementation, and they're complaining that their being involved in this is apparently a waste of their time, and something IT should be able to sort out without bothering them. </facepalm>

      1. James 51

        Re: SO VERY TRUE!

        I find that it is a perceived transfI find that it is a perceived transfer of responsibility that can be responsible for the lack of user engagement. If they don’t engage, then nothing is their fault and if they don’t have to do anything new such as push a button that someone else use to push for them, then they are not responsible for what happens either.er of responsibility that can be

      2. Peter2 Silver badge

        Re: SO VERY TRUE!

        Conversely, in other organisations, there is an IT department which involves the end-users in system specification, design, and implementation, and they're complaining that their being involved in this is apparently a waste of their time, and something IT should be able to sort out without bothering them. </facepalm>

        I think this is a serious and unfair misrepresentation of what happens..

        The end users do complain about being involved, but often come around. They then assign you the worst person on their team, who usually cripples their productivity as they need one person to watch what they are doing and another to fix the fuckups. They do this to gain an extra couple of staff while your working with their liaison person.

        They fail to realise that as a result their computer system is effectively designed by this person and then complain it's a disaster when the system is implemented according to their liaisons feedback.

    3. Tom 7

      Re: SO VERY TRUE!

      "But NEVER NEVER NEVER do they let the actual workers see prototypes or ask them if something will actually DO THE JOB"

      If you get the people who actually know what to do with the data involved you may will finish the job. Unless that is you get some MBAs to ask them what's what. If I apply to do an MBA will they actually remove any logic circuits from my brain?

      1. Chris King

        Re: SO VERY TRUE!

        "If I apply to do an MBA will they actually remove any logic circuits from my brain?"

        From what I've seen of younger MBA's, the logic circuits are all left in place, but the course material shorts them all out.

        1. Bucky 2

          Re: SO VERY TRUE!

          From what I've seen of younger MBA's, the logic circuits are all left in place, but the course material shorts them all out.

          I don't think that's a new thing. In college, if you asked someone "What's your major?" and they replied "Business," it always meant you were dealing with an idiot.

    4. usbac Silver badge

      Re: SO VERY TRUE!

      It really can be done right.

      Several years ago, we needed to replace our primary system (ERP/Accounting/etc.). We researched vendors extensively. Once the top three vendors were chosen, we involved each department lead in the demos.

      Once the vendor was chosen, we had the software company send technical people, not sales people, for a week of training/gap analysis. During these sessions, EVERY SINGLE employee that would ever need to use the system attended the meetings and ere allowed to ask any questions they wanted. The software company wasn't terribly thrilled about all of this, but if they wanted to make a six-figure deal on software, they didn't have a choice.

      The result of all of this was that three days after deployment, almost all of our staff was working productively. The up front costs of doing things this way paid off big in the end. The senior management of our company is very supportive of the IT department, and their support helped make for such a successful project. Ti's really too bad management at other companies just doesn't get it.

      We run our company with about 50% of the staff that other similar companies in the same industry would have. We do this through very good systems and good management of these systems. Fortunately for us, ownership of the company knows this. That's why they support IT so much. They see the value, not just the cost.

    5. Anonymous Coward
      Anonymous Coward

      Re: SO VERY TRUE!

      You see, the high-ups know EVERYTHING. The peons know NOTHING.

      Indeed. Just got an email from a manager asking why a staff-level person three levels down from him had access to certain Admin functions.

      Well, it's because HE knows what's actually going on.

    6. collinsl Bronze badge

      Re: SO VERY TRUE!

      > They, in their infinite wisdom, simply have the system built according to whatever fancies they have relating to what the working classes do all day, then dump that system on the workers, leaving them to sort out the total omnishambles management has once again created.

      Dennis, there's some lovely filth down here!

  2. Ole Juul

    Alameda County spent $4.5m on the system.

    And what about the cost in ruined lives. How do you add that into the price?

    1. DNTP

      Re: How do you add that

      "You don't, it doesn't count."

      -The court system.

      "They're all guilty of something anyway."

      -Our heroes in blue

      "This is going to have to be addressed on a case by case basis by defendants who will have to be able to afford expensive lawyers and appeals with no help from us."

      -HARD ON CRIME District Attorney

      "You guys fucked up my life and now it's either crime or starvation."

      -Guy who got felony record and sex offender listing for a misdemeanor

      1. quxinot

        Re: How do you add that

        It's california. Where else would you find such a great example of what not to become?

        1. Version 1.0 Silver badge

          Re: How do you add that

          "Where else would you find such a great example of what not to become?"

          Trump Tower, New York?

        2. veti Silver badge

          Re: How do you add that

          It's california. Where else would you find such a great example of what not to become?

          America. With the UK, as always, hot on its heels.

          Sad but true.

      2. Steve Davies 3 Silver badge

        Re: How do you add that

        Where are the Shysters when you really need them?

        I would have imagined that the class actions would have been flying now. The county should be on the verge of going Chapter 11. Yet this and the other reports I've read don't mention this at all. Can you sue the Legal System? IANAL etc.

        Meanwhile the people who made the decision are off somewhere else wreaking the same sort of havoc.

        That's life ain't it?

      3. Anonymous Coward
        Anonymous Coward

        Re: How do you add that

        ""They're all guilty of something anyway."

        -Our heroes in blue"

        Shouldn't that read "we know it cost a lot more money, that's why we've been executing so many before they can make it to the court system"?

    2. Oblamo BinLyen

      Re: Alameda County spent $4.5m on the system.

      In Kommieforniastan they don't care if people are hurt, as long as they make it look like they did something good to get the raises and bonuses.

    3. Tom Paine

      Re: Alameda County spent $4.5m on the system.

      This is how we do it: http://researchbriefings.parliament.uk/ResearchBriefing/Summary/SN02131

      (Hmmm, not very well, it transpires. But at least nominally you'd be able to get some sort of financial compensation.)

  3. phantomsteve

    Wouldn't it be a shame if the CIO was 'accidentally' entered into the system as a sex offender....

    1. Antron Argaiv Silver badge
      Thumb Up

      "Oops! It seems our test database accidentally got released with every copy of the product"

  4. Gene Cash Silver badge

    No US coverage of this

    It's interesting how this has been swept under the rug. There was no non-local coverage at all by American news outlets. The only place I've seen it is on El Reg and the BBC (http://www.bbc.com/news/technology-38153992)

    1. William 3 Bronze badge

      Re: No US coverage of this

      Of course not, the US media are too busy implying the Ohio murderer was the victim, and that anyone who didn't vote for Clinton are racist Nazis to bother with actual news.

      1. Anonymous Coward
        Anonymous Coward

        Re: No US coverage of this

        Not voting for Clinton was fine. It was voting for Trump that confirmed those people were a racist Nazis.

        <duck>

        1. Destroy All Monsters Silver badge

          Re: No US coverage of this

          Racist isolationist Nazis, please!

          Also, useful idiots for P.U.T.I.N. whose world-takeover plans are reaching high maturity levels as ISIS-provided freedoms of Syria are tumbling!

    2. Antron Argaiv Silver badge

      Re: No US coverage of this

      It's interesting how this has been swept under the rug.

      If there's one thing politicians and managers are good at...

      1. Ken Hagan Gold badge

        Re: No US coverage of this

        "If there's one thing politicians and managers are good at..."

        ...then I've yet to hear of it.

    3. Anonymous Coward
      Anonymous Coward

      Re: No US coverage of this

      IEEE's magazine, Spectrum, used to have articles on these government IT disasters all the time.

      Seems you would seem the same bad players all the time.

      Did you know they took the ACA systems live without sending ONE registration successfully through the system in testing?

      Spectrum doesn't put those articles online anymore - used to be a weekly feature.

  5. John Smith 19 Gold badge
    FAIL

    Oh dear, the Swiss army knife effect.

    Yeah it's got umpteen dozen features, if you can remember which one to use and where it's at.

    The SAP of courtroom systems?

    Although how many times have the UK courts systems started to computerize?

    How many times has it failed?

    1. Tom Paine
      FAIL

      Re: Oh dear, the Swiss army knife effect.

      Although how many times have the UK courts systems started to computerize?

      How many times has it failed?

      Aer you thinking of NOMS? Libra?

      http://www.computerweekly.com/feature/NHS-courts-5bn-IT-disaster-by-following-the-Libra-route

      Only one icon will do...

    2. You aint sin me, roit
      Facepalm

      Not quite the Swiss army knife effect...

      It's all about the expectations of those silly users...

      "there will be gaps and they're not going to get exactly what they had before"

  6. Anonymous Coward
    Anonymous Coward

    As if the justice system in the US wasn't fucked enough already...

    With cops shooting black folk, a corrupt plea bargain system, a for-profit jail system and now even the court admin is screwing people over.

    Thank fuck the US is a long way away....

    1. druck Silver badge
      Unhappy

      Re: As if the justice system in the US wasn't fucked enough already...

      The US is only a week extradition treaty away...

      1. John Smith 19 Gold badge
        Happy

        "The US is only a week extradition treaty away.."

        Au contraire, with a good legal team it can be a decade away.

        But Blair's little love gift to Shrub is still in force

      2. BebopWeBop

        Re: As if the justice system in the US wasn't fucked enough already...

        weak I think you meant to say?

    2. collinsl Bronze badge

      Re: As if the justice system in the US wasn't fucked enough already...

      To be fair to the US police just look at how many of them have been shot this week alone! 5 in the last 72 hours and 3 in the last 48.

      And the police who have been shooting people do tend to be shooting people who are either pointing guns at them or are waving them about in a frantic manner, or are doing similar with other deadly weapons like knives.

  7. Tom Paine

    Tony Collins....

    ...wrote the book on this.

    https://www.abebooks.com/servlet/SearchResults?an=tony+collins&sts=t&tn=crash

    Dated case studies from the 80s and early 90s but still absolutely relevant. HIGHLY recommended for PMs everywhere. I've bought (second hand) copies for quite a few work colleagues over the years.

    See also: https://www.abebooks.com/servlet/SearchResults?sts=t&tn=normal+accidents

    1. John Smith 19 Gold badge
      Unhappy

      "Dated case studies from the 80s and early 90s but still absolutely relevant. "

      And yet despite the catalogue of fu**ups to study people keep managing to do it.

      Or as the introduction to Peopleware put it

      "Somewhere today someone is writing a Purchase Ledger system from scratch, and failing."

      Of course 35 years later that statement cannot possibly still be true, can it?

      Can it?

      1. Ken Hagan Gold badge

        Re: "writing a Purchase Ledger system from scratch"

        Almost certainly true, because none of the existing COTS offerings will run on hand-held devices and slurp contextual data from social media sites and stash it all in the cloud.

  8. ArrZarr Silver badge

    Brings a whole new dimension to punishment

    Well, Mr Anderson, your crime was stealing a twinkie. Due to an administrative error, you've been booked in for an amputation which you must attend by law. Have a nice day.

    1. hplasm
      Thumb Up

      Re: Brings a whole new dimension to punishment

      Reminds me of this, Mr Buttle...

      http://www.dave.rainey.net/calendars/dystopias/process3.html

      1. Peter2 Silver badge

        Re: Brings a whole new dimension to punishment

        "Well, Mr Anderson, your crime was stealing a twinkie. Due to an administrative error, you've been booked in for an amputation which you must attend by law. Have a nice day."

        That's nothing, you need manual systems to go really badly wrong. There was a case back in Roman times where a man was accused of murdering a sailor and sentenced to death. At the execution the chap he'd apparently murdered turned up and said he wasn't dead, only drunk on the day of the trial. The presiding centurion decided that he wouldn't execute the accused, but refer the matter back to the court.

        The judge at the court flew into a rage at his instructions not being literally and figuratively executed, and ordered the execution of sentence be carried out on the accused, as he'd been found a murderer and sentenced to death. He then had the centurion executed for not carrying out his orders (as he had no authority to stop the execution) and the sailor executed for being responsible for a miscarriage of justice, the death of the two people executed as a result and bringing the courts into disrepute.

        It's supposedly a true story, but impossible to prove given the state of Roman record keeping.

        1. Alister

          Re: Brings a whole new dimension to punishment

          but impossible to prove given the state of Roman record keeping.

          Ah, so they were using Odyssey as well!

          1. MrZoolook

            Re: Brings a whole new dimension to punishment

            "It's supposedly a true story, but impossible to prove given the state of Roman record keeping."

            "Ah, so they were using Odyssey as well!"

            The judge said everyone involved was to be exonerated... Odyssey's poor data entry layout caused some peon to accidentally put 'executed'.

            1. veti Silver badge

              Re: Brings a whole new dimension to punishment

              It's those drop-down menus again. You're trying to scroll down to "Released with a donation from the poor-box", but that just happens to be right below "Crucifixion", and the touchscreen has never been very well aligned so that's that.

          2. Captain Badmouth
            Headmaster

            Re: Brings a whole new dimension to punishment

            "but impossible to prove given the state of Roman record keeping.

            Ah, so they were using Odyssey as well!"

            When I were a lad, the Odyssey was a Greek epic poem by Homer ( no, not that one).

    2. tfewster
      Facepalm

      Re: Brings a whole new dimension to punishment

      Gordon R Dickson predicted this in 1965 in his short story "Computers don't argue"

      https://en.wikipedia.org/wiki/Computers_Don't_Argue

      Edit: hplasm already linked to the full story. Double --->

  9. Version 1.0 Silver badge

    "Public Waste"

    In the US this will be held up as a perfect example of Democratic waste and government overspending by Obama - ignoring the fact that the company selling and installing the software is out of Republican Texas. It's all stupid anyway - this isn't Republican or Democrat - it's simple human greed.

    It's the American way, "Hi Mr Redskin, nice country you got there, we'll buy it for some beads"

    1. Rich 11

      Re: "Public Waste"

      we'll buy it for some beads

      "And we'll give you these blankets for free."

  10. Gordon Pryra

    It has nothing to do with software roll out

    These kind things never do.

    Replace the words "software roll out" with "excuse to move public money to your pocket via your mates company"

    I have worked in multiple local councils in the UK and the problems with these roll outs can ALLWAYS be pointed directly at theft

    No training will be given because there is no money for training, despite paying 5 million. Out of the 5 million, only a hundred k or so will have actually been spent on the system itself, the rest on white elephants

  11. SVV

    Highly configurable

    "Not that the software is inherently bad. It is very capable and expandable. It is highly configurable"

    Software that is described this way is usually inherently bad, not in the sense of it not working, but in the sense of trying to be so flexible to everyone that it becomes inefficient to use for everyone. The description of pages full of drop down menus sounds about right for what happens when this sort of thing is attempted.

    One of the main causes of failure when implementing new softawre systemws is by trying to make everything "generic" and configurable, rather than just coding for what is actually required. I can remember one place where the obsession with making everything generic resulted in systems that were hopelessly convoluted and a nightmare to maintain, and not surprisingly had user interfaces which were unlike anything I had ever seen. Thankfully I was only doing consulting work there as they wanted to move to a web basedf system, so I did at least try to persuade them that doing thje same "genric" stuff was really, really difficult and expensive on the web, I still wonder whether they listened or went ahead anyway, but thankfully it was only a short consulting gig and I was pleased to get out of there with nothing worse than a few amusing stories for my colleagues.

    1. Derezed

      Re: Highly configurable

      Not sure every SME has the resource to employ a full team of software developers straight out of university to reinvent the wheel with slightly rounder spokes hence the market's need for reasonably generic, highly configurable software. Just coding for "what is required" is definitely for smaller applications and not things like ERP systems...or state management of the court system (no, you can't do that in Access). Writing exactly what is needed from scratch when you're not a software company is a risky business...but then again, if the client is big enough and the software house small enough you can pretty much dictate the roadmap of the product...this is just your bog standard implementation fuck up right here.

  12. Anonymous Coward
    Unhappy

    Hold on!

    A management rush, complete with budget.

    A software company that seduced the higher-ups on how great the system was.

    Wide-open configuration with a large number of people given input.

    Inadequate consultation with the people using the current system.

    Lack of training for those expected to use the new system.

    A bureaucracy that won't admit failure.

    Have you been reading our internal purchasing guide?

  13. N2

    Your list

    Isnt that on P1 of the 'projects in controlled environments' manual?

    P2 states 'admit nothing'

  14. Stevie

    Bah!

    I wonder how much money was set aside for the inevitable raft of class-action suits to be filed on behalf of those victimized by this visit from Mr Cock-Up.

  15. Nunyabiznes

    Court software and Tyler

    We have several Tyler modules in our county, and they usually work pretty well. The only limitation is the users generally (either through lack of training or lack of training retention). I doubt the blame for this can be placed on Tyler, at least not the majority of it anyway. We don't use the court module though so maybe I'm completely wrong. :shrug:

  16. MrZoolook
    IT Angle

    Just saying, but, however bad the software is...

    If only there was a way that the people responsible for physically recording the judgements handed, to actually do their fucking jobs, instead of just telling some speccy zitty 14year old trainee clerk to do it.

  17. Anonymous Coward
    Anonymous Coward

    If the software (i.e the courts-module of Tyler) can be shown to be working well elsewhere, than the chances are that it was not just the implementation gone wrong, but the "nay sayers" have realized that they can torpedo a system that will reduce the power of the nay-sayers.

  18. Troubleman
    Facepalm

    Put the money in the bag!

    This sounds very familiar: https://www.youtube.com/watch?v=ImZCXn5Xvso

  19. sgriff35

    Wait, it gets better

    It would appear the Judicial Council that Mr Oyung is now CIO of is very proud of his balls up!!

    http://newsroom.courts.ca.gov/news/2016-distinguished-service-award-honorees-announced

    2nd picture from the left

  20. Dalkeith

    To be fair public sector does specialize in procedures that have the most convoluted flows which can change overnight on political whims. Russian Visa system for independent travelers comes to mind.

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like