Ontario’s AI Note-Taker Accuracy Fixes Start at the Next Purchase

← Back · ACT II · Chapter 9: The Value Control Loop

In May 2026 the Auditor General of Ontario published a performance audit titled Use of Artificial Intelligence in the Ontario Government. Part of it covers AI scribes, the systems doctors and other clinicians can use to write up patient visits. Supply Ontario, the province’s procurement agency, keeps a list of approved systems that health-care providers can buy from, and when evaluators read the notes those systems produced during the bid evaluation, every one of the 20 approved systems the audit reviewed showed inaccuracies. The list today names 28 vendors; the audit covers 20, and neither source says how the remaining vendors were qualified.

The detail matters, because even a lenient test caught the errors. Supply Ontario sent each vendor the same two simulated patient recordings, the vendors generated their notes offline, which the audit says left room to process the recordings more than once, and medical professionals from OntarioMD and Ontario Health reviewed the notes each system generated. According to the audit, notes from 12 of the 20 approved systems “captured a different drug than what was prescribed by the doctor,” nine of the 20 “fabricated information,” and “all 20 of the approved vendors showed one or more of these types of inaccuracies.”

Those results were available to the evaluation. What decided their weight was the points table. The audit reports that the “Accuracy of medical notes generated” criteria accounted for 4% of the maximum score, while domestic presence in Ontario carried a weighting of 30%, and that no minimum passing score was set for accuracy. Supply Ontario disagreed that the weighting was too low, saying “The combined weight between mandatory criteria and technical criteria is appropriate” for security, privacy, bias and accuracy. On where patient data is kept, the evaluation “relied solely on vendors’ confirmation that the data is processed and stored in Canada.”

The Accuracy Fixes Start With the Next Purchase

The auditor recommended raising the weight given to accuracy and setting minimum passing scores. For a buyer, the more useful question is when each fix takes effect.

Recommendation 5, which asks Supply Ontario to raise the weighting for security and privacy controls, bias and accuracy and to set minimum passing scores, opens with the words “For all future procurements related to AI systems.” Recommendations 8 and 9, on bias testing and live demonstrations, open the same way. Supply Ontario disagreed with raising the weighting and agreed to minimum thresholds “for future AI-related systems,” which its response describes in terms of security and privacy controls. Recommendation 6 is not limited to future purchases. It asks for controls that make each user confirm they reviewed the notes. Supply Ontario agreed, and its response commits to study whether that is feasible “in future AI Scribe procurements.”

Supply Ontario’s responses do name one term in the contracts already in force. It says it “has included the annual provision of SOC 2 Type 2 reports in all current AI Scribe contracts.” A SOC 2 Type 2 report is an independent auditor’s opinion on a vendor’s controls over a period of time. It covers a vendor’s security and operational controls; it does not check whether a clinical note records the right drug.

The auditor did not recommend re-testing the accuracy of the systems already approved, and none of Supply Ontario’s responses to the five recommendations on AI scribes (5 to 9) offers to. A separate recommendation, on the public service’s own vendor AI systems, drew a different kind of answer: the Ministry of Public and Business Service Delivery and Procurement says it is continuing to work with partners to “enhance assurance and evaluation practices to ensure testing and monitoring approaches reflect Ontario’s demographic diversity” and to keep accountability “across the AI system life cycle.” The testing the scribe responses add, vendor bias results and possibly live demonstrations, happens at selection, in future procurements. The arrangement the approved systems sit on took effect on April 27, 2025, for “a term of three (3) years, with one (1) additional optional extension term of one (1) year.”

The Live Contracts Carry a Security Check

I do not read this as carelessness. An audit looks back at a process, so most of its recommendations improve the process, and a buyer that accepts them changes how it will buy next time.

Supply Ontario points to a recurring term in every current contract, and the term it points to is a security report. Its published overview of the key evaluation criteria says a solution “must accurately transcribe,” and the evaluation asked whether each vendor monitors its own accuracy. What none of the commitments adds is an accuracy check that Supply Ontario, the agency that approves the list, runs itself on the systems in use now.

That leaves accuracy where it already sits, with the clinician. OntarioMD’s guidelines direct doctors to review the notes manually, and the attestation Supply Ontario may study for future purchases would make them confirm it. Reviewing every note is good clinical practice. It also hands the job of catching the product’s errors to the person the product was bought to save time for, one note at a time, with nothing in the recommendations or responses that turns those catches into a record, held by the agency that approves the list, of how each system is performing.

This is the same gap I have written about in the European procurement proposal, where the care goes into scoring the bid and much less into checking what arrived. Ontario is unusual because the bids were tested, and the test found the problem.

What a Buyer Can Write Before Signing

The useful test from Ontario is whether a fix reaches the contract you are about to sign or only the purchase after it. For an AI system, settle it in writing before signature. The buyer states the property it is paying for in terms it can check (here, that a note records the medication that was actually discussed), the test it will rerun and how often, who runs it, and what happens to a vendor’s place on the list if results fall below the line the buyer set. That is the core of a Request for Value: the buyer writes down the result it is buying and how it will be checked, before any vendor’s proposal frames the question.

A vendor whose system gets the drug right has good reason to want that term. A recurring test turns one offline test on two recordings into a running record that sets it apart from the systems that did not, and the buyer has that record in hand when it decides whether to keep the vendor on the list.

Keep reading: ACT II · Chapter 9: The Value Control Loop