A Post-Implementation Review That Never Asked If the System Worked

← Back · ACT II · Chapter 9: The Value Control Loop

Australia publishes its federal laws on one website. The national auditor describes the Federal Register of Legislation as “the authorised whole-of-government website for Commonwealth legislation and related documents,” and more than 70 agencies contribute content to it. The Office of Parliamentary Counsel, which reports having 135 staff, maintains it.

In 2017 the office planned a new version for January 2020 at a cost of $4.2 million. It went live on 1 January 2024 with a revised estimated budget of $11.8 million, four years late and 181 percent over the first figure. In December 2025 the Australian National Audit Office reported that the office’s procurement and contract management of the project was “partly effective” and that the office “did not demonstrate the achievement of value for money.”

Overruns on government IT are common enough that those numbers would not be worth a post by themselves. What the office did after launch is the part a buyer should look at.

The Review That Was Planned

The office did plan to look back. Its 2017 project framework called for a post-implementation review, which in the auditor’s words was to document “a lessons-learned study that was to be undertaken at the conclusion of the project.” A post-implementation workshop was held in June 2024, about six months after go-live.

The audit report gives its outcome in one sentence: “Against the background that the register redevelopment had been successful, four priority actions were identified”. The four were a project management toolkit, changes to the steering committee’s meeting frequency and decision-making (establishing the committee was assessed as working well), a mid-project health check for large, high-impact projects, and better stakeholder communication.

All four are reasonable, and each is about how the next project should be run. That is what a lessons-learned study is for, and it is what the 2017 framework asked for. None of the four is about whether the new register does what it was bought to do, and the report does not say how the judgment that the redevelopment had been successful was reached, or what it was measured against.

To be fair to the office, the register is in service. Agencies contribute to it and the public reads the law on it, and the office told the auditor in December 2025 that it had averaged more than 20,000 users a day over the previous 90 days. The office also did not wait until the end to look at the project. The auditor notes eight internal progress reviews between 2018 and 2023 and a project health check in 2021. What the report says of them concerns delays, budget and governance.

A system in service is a real result. It is also the result any delivered system would show, at any price and on any timeline.

What The Auditor Found Instead

The audit asked different questions: whether the office’s procurement and contract management complied with the procurement rules and demonstrated value for money, and whether the contracts were managed to achieve what each procurement was for. Twelve of the thirteen procurements involved competition, which the auditor said put the office in a strong position to demonstrate value for money. The auditor found that position undermined by how the procurements were conducted. For ten of the thirteen contracts, “the claimed value for money of the proposed contract price was not evidenced” by a benchmark rate or by analysis of the other suppliers’ prices. Nine of the contracts were varied 50 times, with the variations worth $7.5 million in aggregate.

The finding closest to this subject is about what the contracts asked for. For twelve of the thirteen procurements, the auditor found that clear performance requirements and a means to measure performance were not established. Three had no written contract, and the contracts for nine more did not include performance requirements. Those nine were work orders under a government panel whose master agreement carried general standards, such as supplying services to the buyer’s reasonable satisfaction. The exception was the largest contract, for cloud hosting, which set service levels. Six of the contracts did set out ways to monitor the supplier’s work, such as reports and meetings.

So the examination of value for money that the report describes came from outside the office, and its subject was procurement and contract management. Nothing the report describes, in the workshop’s four priority actions or in the audit’s own scope, weighs the register itself against what it was bought to do.

Why The Buyer’s Own Review Ends Up Here

The gap, as I read it, is one of capability. A review held after launch, inside the organization that ran the project, is well placed to ask how the work went. It has nothing to test the purchase against unless someone wrote down, before the money moved, what the purchase was supposed to produce in terms a later review could check.

The office did write things down early. The auditor mentions a documented technical scope from 2016 and, among the 2017 planning papers, a two-page business plan and a five-page initial project plan, the one carrying the release date and the budget. A technical scope says what will be built. The report does not say whether any of these papers stated what the register was supposed to produce in terms a review could check. It does say that as delays occurred and costs increased, the 2017 documents “were not updated or revised.”

The question itself is short. New Zealand’s government procurement guide carries it in its end-of-contract review, in six words: “have the anticipated benefits been received.” A buyer can answer that only if the anticipated benefits were recorded somewhere the reviewer can find them, in the buyer’s own words, with a way to tell whether they arrived.

Without that record, the review falls back on the questions it can answer. Was the project well run? What should we do differently? Those are useful questions about the team, and they leave the purchase unexamined.

What A Vendor That Delivers Gets From This

A supplier that brings a project in on time, on budget and doing what was promised has the most to lose when the buyer’s review has no stated result to measure against. A clean delivery and a late one can end up described with the same word, and the vendor that actually produced the result has nothing in the buyer’s record that sets its work apart from anyone else’s.

A buyer-held statement of the expected result changes that. The buyer writes down what the purchase has to produce and how that will be checked, the vendor answers in those terms, and the buyer keeps both for the review. That is the job of a Request for Value: it gives the review something to ask besides how to run the next project, and it gives a vendor that delivered something to point to.

Two of the auditor’s eight recommendations come closest to this. The first asks for timely and comprehensive project planning, including an assessment of project risks and of the office’s own capability to undertake more complex procurements, so that the next version of the register can be delivered on time and within budget. The eighth includes establishing a written contract “with performance requirements and a specified approach to measuring contractor performance, before a contract commences.” The office agreed to all eight. Neither of the two asks, in terms, for a statement of what the next version itself is supposed to produce that a later review could check, and that statement is the buyer’s to write.

Going back after delivery is the step described in Most Companies Set Goals for Purchases and Never Check Them and in Chapter 9 of The Value Standard. United States federal buying has a written annual review that also leaves out what the purchase produced, covered in The Annual Review Is Going From Three Questions To Six. A different Australian audit, where every delivery metric read complete and the auditor could not verify the upgrades, is in An Australian Audit Found Upgrades Marked Complete but Unverified. New Zealand’s annual procurement capability report, which counts how many agencies recorded on-time and on-budget delivery and nothing about results, is covered in New Zealand Tracks On-Time, On-Budget Purchases but Not Results.

References: Australian National Audit Office, Procurement and Contract Management of the New Federal Register of Legislation Project, Auditor-General Report No. 15 2025-26, December 2025 (summary paragraphs 11, 12 and 22; paragraphs 1.1, 1.3, 1.6, 1.7, 1.11, 1.12, 2.4, 2.5, 2.7 to 2.9, 2.47, 3.8, 3.16 and 3.20; footnotes 8, 21 and 54; Recommendations 1 and 8). New Zealand Government Procurement, Guide to Procurement: Conducting Reviews.

Keep reading: ACT II · Chapter 9: The Value Control Loop