Learning from the Qld Health payroll fiasco

The torturous implementation of Queensland’s Health Payroll system will almost certainly be remembered as one of the most disastrous IT projects in our country’s history.

What began as a $6.19 million contract between the State of Queensland and IBM Australia to replace QLD Health’s aging payroll system eventually led to over 35,000 payroll anomalies and will ultimately cost taxpayers $1.25 billion.

While the final autopsy of the project is yet to be concluded, the project has been subjected to numerous reviews and a state level inquiry that saw a number of key IBM and QLD State Government personnel take the stand.

While we will have to wait until the end of July to see the report from the Commission of Inquiry, the public hearings have so far provided sufficient glimpses of how and why this project went so far off the rails.

Troubled beginnings 

There were early signs of inadequate internal governance and mismanagement well before the QLD Health payroll project was awarded to IBM in December of 2007. The Queensland Treasury had spent the preceding two years "burning through the budget" by attempting to implement a standardised SAP-based HR and finance system across the whole of government operations - the very same platforms that were picked for the Queensland Health payroll project.

To aid in the software development and implementation, Queensland Treasury hired a large number of contractors from a mix of external consulting companies including Accenture, IBM, Logica and SAP.

Former managing director of Accenture, Marcus Salouk, described the state government progress between 2005 and 2007 as “burning through their budget without getting commensurate outcomes”.

“The Queensland Treasury were not experienced system implementers and in my observation, not professional project managers. I was concerned that going through that process, they would end up with insufficient budget to actually get where they needed to get to with implementing the new system”, Mr Salouk told the inquiry.

After gaining little headway, the state government decided to relinquish its role as a systems implementation lead to a single prime contractor.

The role was awarded to IBM in December 2007 with CorpTech, the specialised business unit of the Treasury, entrusted with the responsibility of managing the prime contractor, IBM.

The delays suffered during the wider state rollout of the SAP-based system had already highlighted that the departments within the state ,including Queensland Health, were not ready for a standardised system headed up by an external prime contractor, as it would require the government to detail its specifications with a great amount of accuracy. 

Former program director of CorpTech, Darrin Bond, said in a witness statement that just prior to IBM being awarded the prime contractor role, the departments within the state “were still debating and arguing about what they would or would not get and what they would and would not accept”.

In the absence of the internal requirements being agreed upon for this government-wide system, the scene was set for a prime contractor to come in under a continually varying scope environment and ultimately leading to a project blowout of both time and cost.

The writing was already on the wall.

A gross underestimation

The stunted progress of the implementation project that the state had spent two years trying to rollout had a direct impact on Queensland Health, who were originally scheduled to receive the new SAP-based system in 2006 and were using a decade-old LATTICE payroll system at the time.

With the supplier ending support for the LATTICE payroll system in July 2008, a decision was made in late 2007 by Queensland Health and CorpTech to escalate the implementation of the Queensland Health payroll system. That’s where IBM came into the picture, in what would be its first project as a prime contractor role. 

The contract price negotiated for the design and implementation of the new payroll system was $6.19 million and was to be delivered in July of 2008 - the same month that the support for the existing payroll system was scheduled to expire.

The fact that Queensland Health, CorpTech and IBM agreed to a seven month timeframe to deliver a payroll system with complex award structures that spanned 13 awards and multiple industrial agreements, and contained in excess of 24,000 different combinations of pay for 80,000 employees shows a gross underestimation of the project at hand.

What is even more alarming is that according to the original project documentation, a timeframe of only two weeks was allocated to determine the business requirements and solution scope of the complex payroll project.

To put this in context, a similar payroll project took 12 months to scope and three years to rollout at the privately run Mater hospital, which incidentally finished on time and on budget.

It’s no wonder then that the Queensland Health project soon started to groan under the weight of scope creep, with a recorded 47 submitted change requests signed off by CorpTech in a space of just two months after the contracts were signed.

As a result of the poorly defined business requirements cobbled together at the beginning of the project, several witnesses questioned during the inquiry spoke of ongoing uncertainty about what exactly the project was required to deliver and a project scope that remained open-ended throughout the life of the project.

Cutting corners and IBM’s duty of care

While the time-strapped state government was allegedly responsible for setting the timeframe, it almost certainly raises a duty of care issue on IBM’s part.

Independent expert, Dr David Manfield, who was engaged by the Commission of Inquiry, said in his report that IBM must have recognised the complexity and size of the solution required well before they were awarded the prime contractor role.

“IBM must have known the number of awards and the known existence of many business rules governing payroll and the likely difficulty in determining the business requirements and solution scope in such a short amount of time,” Dr Manfield said.

“IBM must have been fully aware of this risk, in particular as it had an existing relationship with QLD Health.”

However, IBM continued to operate under the varying scope and the state government kept signing off on the change requests. By the time the system finally went live in March of 2010, 20 months after the original start date, the bill had already ballooned to $101 million. 

The project documentation also reveals that prior to the payroll system going live the project underwent four revised go live dates and four separate stages of change requests, often done at the last minute.

The delayed establishment of a mutually agreed baseline scope impacted every aspect of the project including the implementation and testing phases.

Under the pressures of a time imperative, substantial corners were cut and according to Dr Manfield’s report, the state government intentionally lowered the bar for testing and knowingly allowed a flawed system to go live.

The cost of going live with a premature system resulted in more than 35,000 payroll anomalies and to date has cost the state in excess of $400 million, just to operate the system. KPMG has estimated that the cost of making the system function for the next five years will be another $836 million.

Questionable procurement processes

One of the areas closely scrutinised by the inquiry has been the procurement processes that led to IBM being appointed as the prime contractor. It’s certainly an area worth examining.

The fact that a former IBM executive, Terry Burns, had been tasked with running the bidding process, should have immediately sent alarm bells ringing over the credibility of the entire procurement process

Burns has since admitted to having let IBM have a “dry run” of its bid for the prime contractor role, a luxury not afforded to the other bidding vendors, Accenture and Logica.

The inquiry has also revealed the leaking of a competitive bid and alleged modification of scoring during the final selection process.

Witness statements from certain members of the selection panel allege that Accenture was the front-runner until Burns urged the panel to re-evaluate the submissions, which ultimately swayed the vote towards IBM. Burns denies that he unduly favoured IBM, while CorpTech maintains that they wanted to encourage as much competition as possible between vendors.

The final say on the matter will come from the leader of the Commission of Inquiry, Richard Chesterman QC, but there is little doubt that the  procurement process was hopelessly compromised and it’s now just a matter of pinpointing when the rot set in.

Lessons learnt?

Questionable procurement practices aside, delivering an ICT project of this complexity and scale is not a joke. And the task was made harder by basing implementation on loosely defined business requirements within an impossible timeframe. A project blowout was always on the cards.

The system failed critical user acceptance testing (UAT) processes but instead of addressing the issues, the bar for testing was simply lowered and less stringent guidelines adopted in an effort to get the system out the door to meet a time imperative.

There is little doubt that the state government should have acted to address the underlying issues instead of ignoring the risks and letting a flawed system go into production. A system that left thousands of Queensland Health employees underpaid, overpaid or not paid at all. 

The Queensland government obviously isn’t alone when it comes to government run ICT project blowouts. New South Wales had the failed Tcard project, while the Victorian government botched up the deployment of the Myki smartcard systems.

These were all flawed systems that were rolled out prematurely and resulted in significant blowouts at the expense of taxpayers. They also highlight the systemic deficiencies in internal governance, contract management, strong controls over budgets and thorough testing and implementation regimes.

While the recriminations over the $1.25 billion bungle in Queensland will continue long after the Commission of Inquiry hands its final report to the state premier, the real question is if any lessons have been learnt from the big-ticket failure. The taxpayers would certainly hope so lest they end up holding another costly clean-up bill in the near future. 

Krishan Sharma is a Brisbane-based Freelance Journalist and writes for a number of different publications covering Business IT and Consumer Technology.