back to article 'Incomprehensible failure' – Canada's $1bn Phoenix payroll IT fiasco torched by auditors

Canada's top auditor has issued a scathing postmortem report on Phoenix: the nation's disastrous attempt to overhaul a key government IT system. In its comprehensive dossier outlining the nine-year-long fiasco to replace the software that handled the country's payroll system for government workers, the Office of the Auditor …

  1. a_yank_lurker

    Ouch

    How many screwed up IT projects suffered from similar issues before being put of their misery?

    1. Anonymous Coward
      Anonymous Coward

      Re: Ouch

      The problem with these sort of "replace something we've been using for decades" projects is that no one really knows what the current system does. It has been tweaked and tweaked over the years as issues were found and worked around, but they weren't properly documented as to why they were done.

      When you have people spec'ing the new system, they what it does on a day to day basis and the requirements capture that, but the thousand exceptions that have been built up over the years aren't and that's where they always fall down.

      I have no idea how to solve that problem, and I don't see how it can be solved to be honest. All you can do is develop the new system and live with the pain while you build up the same capabilities to handle all those corner cases - and make damn sure you document them well so the next time you rip and replace your requirements capture 99.98% of what needs to be done instead of only 98%.

      1. yoganmahew

        Re: Ouch

        @DougS

        As someone going through this again (I'm on the 50 year old application side), part of the problem is the unwillingness to actually pay for the expertise in the old system to oversee the functional design of the new system. It's less that the old lags who keep the old system running and up to date aren't willing, more that the minion-driven-development in the outsourcer doesn't want to pay the cost. Meanwhile, the old lags watch on with horror as 50 years of mistakes are repeated...

      2. Giovani Tapini

        Re: Ouch

        Good points, but it is not clear that this is the issue in this case.

        It seems more like a lot of de-scoping anything that might be "a bit hard" as the budget and delivery deadlines were enforced as if set in stone.

        That way you get the delivery party while the rest of the organisation wonders how they can operate and work 24x7 to keep things moving via workarounds.

        This is not unusual. It is more likely where projects are primarily run by third party integrators too. They have no incentive other than the contract which will largely define budget and timescale rather than any useful attributes of the deployed solution.

        I have experienced the above multiple times now in my career across different scale projects. It usually stems from the translation of guesses into a project plan with little or no proper analysis phase (which can look expensive) to more effectively define and scope what actually is supposed to be delivered.

      3. Christoph

        Re: Ouch

        I was once involved in specing a computer system for a department of about 15 people, all in one large office. We got everyone together to sort out what the new system would do to replace the current method of working.

        It emerged from that meeting that nobody in the office actually knew exactly how the current system worked.

      4. Yet Another Anonymous coward Silver badge

        Re: Ouch

        is that no one really knows what the current system does

        And the supplier knows this.

        In spite of 6000 page specifications in this case - there will be things missed.

        The suppliers bid to lose money on the original contract and then make the real profit on billing vast amounts for any inevitable changes. For extra bonuses you implement things in the original spec wrongly - but legally in keeping with the spec - and hope for big $$$$$ to fix it

        1. Daniel 18

          Re: Ouch

          Am I the only one to suspect that a 6,000 page specification is too long to be a useful specification document?

          Would anyone read, comprehend, and remember the whole thing? Would any group of people doing 'specification level' analysis be able to discuss it adequately in a week?

          A detailed design document, possibly, but even that seems overly complicated for a payroll system, even one that can do retroactive payments.

          1. Yet Another Anonymous coward Silver badge

            Re: Ouch

            So what is the procedure for paying time in lieu to a federal employee of union A required to be on standby on a provincial holiday to supervise an employee of Union B completing a task at a site that qualifies for remote living allowance?

            There are literally 1000s of these - miss one out and IBM will happily charge you $10,000 a day to document, specify and implement the fix.

    2. Alistair
      Mushroom

      Re: Ouch

      @a_yank_lurker:

      How many screwed up IT projects suffered from similar issues before being put of their misery?

      There have been numerous books listing quite a few. I know from personal experience of at least 8 in my 20 year career in current, and in almost all the cases I know of, not being utterly clear on the objectives and requirements has killed each and every one of them by running it out to the middle of the span before there was even a dream of a bridge being built.

      picture Wiley E in that moment, just after he's passed the cliff edge, before he glances down

      Icon is the inevitable result.

  2. John Smith 19 Gold badge
    Unhappy

    top 5 causes for major IT project failure *still* top 5 causes cause of major IT failure

    Good to know.

    And maybe one day the government department of some government will actually apply those lessons before it causes such a massive f**k up.

    Basically they chose "On time and budget" over "does the whole job."

    If storing and processing the information correctly is not a key requirement, guess what, I can guarantee I can deliver your system.

    It won't actually be very good, but it will be on time.

    1. JLV

      Re: top 5 causes for major IT project failure *still* top 5 causes cause of major IT failure

      Hmmm, at $850m and counting, "on budget" wasn't exactly a high-priority priority they achieved either ;-)

      They should have tested w at least 2 major departments, preferably with 2 different collective agreements/unions underpinning payrolls. This project was supposed to consolidate multiple departments, so even a 1 dept successful pilot might not have unearthed challenges w merging many departments.

      Feds already run PeopleSoft payroll, so messing this up was not a given from the start.

      1. Yet Another Anonymous coward Silver badge

        Re: top 5 causes for major IT project failure *still* top 5 causes cause of major IT failure

        Although for political reasons part of the plan was to shut down the previous dept on mass just before the new system was rolled out - so that no long term employees would be carried forward.

        Can any 5 year olds see a flaw in this plan ?

      2. John Smith 19 Gold badge
        Unhappy

        Hmmm, at $850m and counting, "on budget" wasn't exactly a high-priority..achieved either ;-)

        Well y'know all those "pre-sales" types aren't cheap either.

        I said I could deliver it on time (if it didn't have to work).

        Never said it would be on budget as well.

  3. Howard Hanek

    Vizzini here! Have some iocane powder! It'll solve all those enigmatic problems.........

  4. Oengus

    Universal

    I just worked on a payroll implementation and we had exactly the same issues... management emphasising schedule and budget over function. Management trying to get us to bypass testing for the sake of the schedule and really pushing back when issues were raised. The last thing that the project manager wanted to hear was that something didn't work as expected and the response was always "Can we fix it after we go live?" The big issue still is "retroactive actions". The project manager had us go to senior management to request that they completely ban backdated changes (in this organisation that is never going to happen) because the payroll system couldn't handle it. Even now there are issues with some backdated actions that require manual calculations (when we find out that it has happened) to have people paid correctly.

    1. JustWondering
      Facepalm

      Re: Universal

      Add to all that the idea that this program was customized to automatically enter incorrect information. No matter what salary a person may have negotiated, the program was deliberately designed to enter the lowest possible wage for that classification.

      1. Anonymous Coward
        Anonymous Coward

        Re: Universal

        I'll never forget the statement from a colleague on a payroll project.

        "It's OK, only six people didn't get paid..."

        I didn't really know what to say to that, but looking over these comments it is apparently normal business practise, I must have missed that bit of my training somewhere.

  5. Anonymous Coward
    Anonymous Coward

    This has caused a great deal of pain and grief here in Canada. I sincerely hope we will learn from this. There is a small part of me that also wants some of the Public Services and Procurement department sparta kicked into the void.

    1. Nolveys

      There is a small part of me that also wants some of the Public Services and Procurement department sparta kicked into the void.

      About 90% of the Canadian federal government could vanish into the void without any meaningful loss of service.

      1. Andre 3

        About 90% of any government could vanish into the void without any meaningful loss of service.

        Fixed that for you

      2. Anonymous Coward
        Anonymous Coward

        And about 70% of most provincial governments.

    2. Anonymous Coward
      Anonymous Coward

      "This has caused a great deal of pain and grief here in Canada. I sincerely hope we will learn from this."

      I doubt it.

      Ontario's E-health system project spent a billion dollars to produce... nothing.

  6. SVV

    Time-Quality-Cost triangle

    As all project managers believe, you can pick two out of the three with a fixed budget, but will sacrifice the third.

    As all developers who work for project managers who believe this know, you will deliiver late, have poor quality and exceed the budget.

    1. trevorde Silver badge

      Re: Time-Quality-Cost triangle

      Worked at a place where management wanted it on time, at high quality and within budget. We dubbed it the 'Iron Triangle'.

  7. Destroy All Monsters Silver badge
    Holmes

    Sigh

    Looking ahead, the auditor general is recommending a set of new requirements for how the Canadian government should handle major IT projects, including the use of internal audits and oversight groups that would help keep a close eye on how projects are being rolled out.

    So you have to have greater, more adult adults to oversee the supposed adults in charge of the project?

    Why won't anyone read the classics? Or apply lessons learnt during the last 70 years?

    Why, I have a 620-page manual on my desk full off good, classic advice: "Software Engineering

    in the Systems Context - Addressing Frontiers, Practice and Education" (Ivar Jacobson, Harold “Bud” Lawson, Eds).

    It's frankly required reading, but of course when you put this in front of the PM or The Decider, he/she will immediately ask "Do you expect me to read this?". Upon which the correct comeback is "No, I expect you to fail". "We need to have this done by X" shall be answered with "You won't". "Failure is not an option" will lead to "It is now". "I have an MBA" can only be countered with "LOL".

    From said book:

    1. The ignorant do not know known good software engineering practice.

    2. The neglectful may know known good software engineering practice but lack the respect and discipline to apply it rigorously as intended.

    3. Those who intentionally defer work or take expedient shortcuts are arrogant practitioners who believe their superior skills will permit them to dodge the bullet of consequences for their action.

    I often hear that PeopleSoft is a hazard zone of well-remunerated dark consultancery that delects to remora onto deep-pocketed, possibly taxpayer-funded organizations that are too big to fail.

    IBM was chosen as the contractor to deliver a system based on Oracle's PeopleSoft system.

    It could be true.

    1. W.S.Gosset

      Re: Sigh

      Upvote.

      >delects to remora onto

      And another upvote for magnificent english.

      1. BebopWeBop
        Joke

        Re: Sigh

        remoras suck - as we all know

  8. Anonymous Coward
    Devil

    > IBM was chosen as the contractor to deliver a system based on Oracle's PeopleSoft system.

    This explains everything. It was doomed from Day One.

    1. Denarius

      sounds like the Queensland Gov fiasco

      same outsourceries in charge also.

      1. Anonymous Coward
        Anonymous Coward

        Re: sounds like the Queensland Gov fiasco

        same outsourceries in charge also.

        The PHBs at IBM wanted a cookie-cutter business model to cheaply replicate their past "successes", albeit at an unchanged cost to customers, and that's what they now have.

        IBM: Guaranteed failure, anywhere, any time, any place.

  9. JustWondering
    Facepalm

    What could go wrong?

    Unfortunately, those making the decisions to go live didn't hear from those who were actually working on the project. They seem to have gone on the basis of what they were told by those hoping to get their bonuses for a timely implementation.

    1. AJames

      Re: What could go wrong?

      And that was the key problem: there were big management bonuses tied to on-time delivery. People are people. They were determined to deem the system delivered on time and get their bonuses no matter what, and they did (and they got to keep them!). Somebody forgot to specify that what they delivered should work.

      You can't imagine the nightmare this has caused for employees incorrectly paid. Some have gone without pay for months and are in danger of losing their homes due to defaulted mortgage payments. Others have been overpaid, taxed incorrectly as a result, and are now trying to fix that error, which will result in more fixes next year, and fixes to the fixes the year after that etc. etc.

  10. Michael Hoffmann Silver badge
    Facepalm

    So?

    What heads actually rolled amongst those to blame?

    Any? Even a single one?

  11. Anonymous Coward
    Anonymous Coward

    There's a point here that nobody mentioned

    Yes, a deployment might be subject to partial or total failure.

    One big question to ask is how comes nobody thought of a roll back plan ? You know, just in case sh&^%$t happens, we go back to what we know was working before.

    1. Anonymous Coward
      Anonymous Coward

      Re: There's a point here that nobody mentioned

      I thought they picked the burnt-bridge fallback by ensuring that the rollout was one-way.

  12. Scott Marshall
    FAIL

    Not the first time this has happened with IBM ...

    Here in Down-Underland, the Queensland Department of Health had similar problems with, yes, a payroll system developed by IBM.

    cf: Learning from the Qld Health payroll fiasco

    ... and ...

    Queensland Health payroll fail: Government ordered to pay IBM costs

    Once again, due diligence by the government agency (or agencies) was not adequately performed (if performed at all).

    Of course, let's not mention the Australian Census debacle. But then again, let's do:

    Tech giant IBM pays about $30m compensation to taxpayers over botched census

    IT disasters now part of modern life

  13. DrM
    FAIL

    Let's just skip that step

    "The Department did not completely and properly test Phoenix before its implementation, which is contrary to recognized practices for developing a system," the audit stated. "Phoenix executives cancelled a pilot project with one department that would have assessed whether Phoenix was ready to be used government-wide."

    In the automotive assembly plant world, this would be going with production tooling and skipping prototype tooling.

    1. Daniel 18

      Re: Let's just skip that step

      "In the automotive assembly plant world, this would be going with production tooling and skipping prototype tooling."

      Or going to production on F35s before making them actually work...

      Is this one of those universally applicable errors?

  14. Guus Leeuw

    Dear Sir,

    this has gone on for ages and ages. And it will continue for ages and ages...

    The problem is two-fold:

    1) Management is paid (as already indicated) partially by bonuses upon delivery of something. That leads to rushed schedules. Always has and always will.

    2) Even when the project is intended to be agile, there's always somebody fixing the budget. That leads to delayed functionality.

    I always ask my clients: Is this an agile project? If they say yes, I present them with a never-ending, all-encompassing contract where they pay me $y a month, and I promise to deliver x hours worth of work doing whatever it is they tell me to do. They always come back with: "But we only have $ amount"... To which I always reply: "So the project is not agile, then, is it?"

    If the clients indicate that it is a waterfall project, I immediately pull out a Business Analyst and let that person have a go at finding all the requirements. Then I add a System Architect / Data Architect to translate those requirements in costable work-units. Then I add a Finance Guru / Sales specialist to come up with a cost for the *agreed* scope. When everything is signed, I add a dogmatic scope-and-finance driven project manager to control the client, whilst I let developers get on with the job at hand.

    If the client wants to restrict time, or start early, we can always talk about versions of the software-to-be-built. The balance the *client* has to find is: what functionality is needed for the first version. A very difficult balance to strike.

    Ultimately, even in properly scoped, documented, managed projects, there will inevitably be delays. What most projects and project managers forget is that you *need* a communications matrix that tells you who your friendly counterpart is to help you solve which type of problem (money, schedule, scope). For *massive* replacement projects, you need to take all the end-users from day one on a journey that tells them about the new system, shows them how it works, and gives key-users the ability to interact with the new system while it is being build... *Never* *ever* develop something in a closed-door environment to reveal it on D-Day to the hordes of end-users... They will inevitably not like what they see, because it is "different"...

    There you have it... More than $0.02 worth, but you're very welcome!

    Regards,

    Guus

    1. Anonymous Coward
      Anonymous Coward

      "2) Even when the project is intended to be agile, there's always somebody fixing the budget. "

      When someone tells you that a project more complicated than a simple web site is 'agile', my observation is that the person telling you this somehow thinks they can avoid doing a proper analysis and design phase with no penalty.

      ,,, probably due to inexperience, inflated self-regard, and a tendency to believe vendors selling 'magic bullets'.

    2. John Brown (no body) Silver badge

      "1) Management is paid (as already indicated) partially by bonuses upon delivery of something. That leads to rushed schedules. Always has and always will."

      Maybe bonuses should be tied to to successful implementation/module stages (50%) and the remaining 50% on meeting the time constraints, the latter being null and void if the former isn't met.

  15. Cuddles

    Priorities

    "Phoenix executives prioritized certain aspects, such as schedule and budget...

    The Trudeau government estimated the total cost to fix the system could hit CAN$1bn by 2022"

    Six years late and well over three times the original budget doesn't really sound as though schedule and budget were prioritised. Cheap, fast, good - pick none.

  16. NBCanuck

    Project

    You want us to deliver "X" . Sure....no problem.

    You want high quality, fast delivery and you want it cheap? Pick two!

  17. Herring`

    I think that if we've learned anything from history it's that we don't learn anything from history.

  18. Justicesays

    Irony

    Just before I left IBM there was a big push at the management level towards "devops/agile"

    So they were all reading this book:

    https://www.amazon.com/Phoenix-Project-DevOps-Helping-Business/dp/0988262592

    I guess it turns out real life is harder than fiction

  19. Anonymous Coward
    Anonymous Coward

    BTW the executives on this project did actually get bonuses for this debacle.

  20. bfwebster

    Not incomprehensible at all, sad to say. Most of the core issues here have been well documented for decades. The problem is that most people in IT (especially upper management) don't read established basic works like _The Mythical Man-Month_ (Brooks) or _Peopleware_ (DeMarco & Lister).

    Failed IT projects has been my major area of professional focus for over 20 years; I've advised major corporations, testified before the US Congress, given lectures, published articles, acted as an expert witness in lawsuits involving failed projects. I currently teach a senior-level (US) university course on software engineering, and the two books above are among the required reading. What is utterly depressing is how predictable most failed IT projects are and how they all typically stem from a handful of well-known root causes, most of which involve a tremendous amount of wishful (but erroneous) thinking.

    1. Anonymous Coward
      Anonymous Coward

      ”Failed IT projects has been my major area of professional focus...“

      Are you an IBM executive?

      1. Anonymous Coward
        Anonymous Coward

        Clearly he is not

        as he actually knows why projects fail and talks to people about how to prevent it.

        IBM on the other hand don't really care if projects fail or not, as long as they can still get paid.

  21. rbf

    Only a government could survive a payroll system collapse

    Way back when, I was given a project to port a number of engineering applications mostly in Fortran from VM to (yes) Multics. So I picked an application guinea pig for a pilot project to quantify the effort that would be required to do the whole lot.

    The vendor touted its Conversion Support Team; so I brought them in for a meeting. In their eyes, my pilot would be a major project - not a good omen.

    The pilot quickly crashed and burned with several mysterious messages from Multics and the compilers. My conclusion was that the technical demands vastly outweighed the available expertise including the vendor Conversion Support Team.

    The conversation project was called off. I heard some talk that I had failed while I had in fact spared the company from a fiasco that would have cost many times my salary over my entire tenure.

    The Multics system was eventually brought in to replace the vendor's earlier Time Sharing System. It gave no end of trouble and I occasionally stuck my head in my manager's office to remind him of the misery I had spared him from.

  22. Anonymous Coward
    Anonymous Coward

    Complexity, puh.

    The basic functionality is only just out of reach of a Spreadsheet.

    Due to widespread reading comprehension issues, I'll restate it again more clearly: The basic functionality *is* clearly out of reach of a Spreadsheet, but not by a wide margin on a Log scale. Perhaps 10:1 ratio, roughly. In the late-1970s, this function might have been coded up in BASIC and run on a TRS-80. The month's data would probably fit onto a 5.25-inch floppy.

    Same folks were presumably involved in the aborted 'Long Gun Registry', another billion-dollar Canada government IT boondoggle. Yes, there is a trend.

  23. Daniel 18

    The history of computing/IT is largely the history of failed 'magic bullets' which were going to make everything easy, solve all the problems, slash costs and delays, and often, eliminate programmers, analysts, IT professionals, etc.

    Some highlights from the list include COBOL, subroutine libraries, time sharing, spreadsheets, relational databases (no need for IT pros getting between end users and their data if you have one of those!), remote computing, GUIs, structured programming languages, gotoless programming, object oriented languages, the cloud, microservices, agile, devops....

    In many cases these fads are driven by a blind faith that a particular technique will solve all the major problems, or a desire to fix irritants that are the result of a solution to a previous problem... which often returns when the new magic bullet sweeps away the technique or practice that eliminated the older problem.

    Often these ideas are 'validated' by anecdotal results achieved by the people who designed and built the new tools or techniques.

    Curiously, random user populations with diverse skills and requirements often under-perform when compared with the developers or researchers using the tools they designed to solve the problems they are working on.

  24. ecofeco Silver badge

    No mystery here

    IBM? Oracle?

    A blind man can see what went wrong here.

  25. Anonymous Coward
    Anonymous Coward

    Phoenix made IEEE's Risk-factor failure list

    https://spectrum.ieee.org/riskfactor/computing/software/2018s-it-failures-have-a-familiar-look-already

  26. DanceMan

    Perhaps an insider can provide more accurate info, but I recall reading that this project was also pushed into implementation by the Harper government anxious to show cost savings on paper to meet their hallowed balanced budget.

  27. Potemkine! Silver badge

    If I got it well

    somebody was required to manage the project managers?

POST COMMENT House rules

Not a member of The Register? Create a new account here.

  • Enter your comment

  • Add an icon

Anonymous cowards cannot choose their icon

Other stories you might like