Why AI Success-Story Collections Can’t Prove ROI for You

← Back · ACT II · Chapter 7: Value Is Not Promised; It Is Proven

Procurement and supply chain leaders are being handed a lot of AI evidence right now. Use-case libraries, playbooks with a hundred or more examples, ROI benchmarks gathered into free guides that arrive in exchange for a comment or an email address. Some of it is useful. If you want to know what other companies have tried with spend analysis, demand planning or freight, a long list of examples saves you time.

The trouble starts when the list gets called proof.

Where the Numbers Come From

Look at where a typical case study’s before-and-after figures come from. Most were published by the vendor that sold the system, by a consultancy that helped put it in, or by a trade article quoting one of them. The buyer’s name is on it, usually with permission and a quote that marketing on both sides approved. The baseline was picked after the project started. The improvement was measured by the party that benefits from it looking large. Nobody outside checked it, and almost nobody went back a year later to see whether it held.

Most of my career went into building the vendor business cases that stories like these grow out of. The customer was usually happy, the number was real in some sense, and the story was worth telling. But it was our story, built from the data that made the strongest case, and a reader had no way to see what had been left out of it.

Collecting Them Doesn’t Change What They Are

Put a hundred of these in one book and the collection looks stronger than any single story inside it, but a hundred results written by the people who sold the systems mostly add up to a very good picture of what vendors like to publish. AI has also made these collections cheap to produce. A model can gather and polish a hundred success stories in an afternoon, with nothing underneath any of them that a buyer could check.

Selection does most of the work. Purchases that disappointed rarely get a case study, and the ones that went sideways after the first year never get a second chapter. So a library made only of successes tells you almost nothing about the odds that your own purchase will become one.

The examples are still worth reading as a list of possibilities, which is how a buyer should use them.

What Would Make One Count

For any of those examples to count as proof, a few things would have to exist that almost never do. The buyer, not the vendor, would have written down before signing what the purchase had to produce. There would be a baseline taken from the buyer’s own records, a date to check it, and a plain statement of what had to be true on the buyer’s side for the result to arrive. And someone on the buyer’s side would have looked again on that date and written down what actually happened, including when it fell short.

Take a common example, an AI tool that cuts invoice processing time. The case study reports the drop. It doesn’t say whether the people who used to process those invoices moved to other work or are still on the payroll doing less of it, whether the exceptions simply landed in another queue, or whether the time saved ever reached a budget line. Those are the questions a buyer would have written into the page before signing, and they are the ones that decide whether the purchase paid for itself.

That record is small, usually one page. It is also the part none of these case studies show you. That’s why the case-study libraries keep getting longer while the question finance asks a year later, whether the thing worked, still has no good answer.

What a Vendor That Delivers Gets Out of It

The vendors with real results lose the most in a market full of borrowed numbers. Their case study sits next to a competitor’s that was written the same way, from the same kind of hand-picked data, and a buyer can’t tell the two apart.

A result the buyer defined before the contract and checked afterward is different. It is evidence the vendor didn’t write, which is the only kind the next buyer has a reason to trust. A vendor that delivers should want that page more than anyone.

Use the Examples, Keep Your Own Record

Read the libraries for ideas, and borrow freely. When one looks like it fits, write your own version before you buy: what this purchase has to do for your company, measured from your baseline, by a date you set, with the conditions on your side written next to it.

The Request for Value is a one-page form for exactly that, and the Purchase Outcome Register is a free workbook that keeps the record across every significant purchase. Both are on the Guides page. The usual objection, that writing this page down slows the purchase, is taken up in Agree on What a Purchase Must Produce Before You Sign.

The case study worth having on your next AI purchase is the one your own company starts writing before the contract is signed.

Keep reading: ACT II · Chapter 7: Value Is Not Promised; It Is Proven