Analyst Ratings Rank Vendors on Criteria You Didn’t Choose

← Back · ACT II · Chapter 11: Outcome Credibility Markets

Market ratings are getting better at telling buyers which providers deliver. None of them can tell a company whether its own purchase did what it was bought to do.

Something useful is happening in how service providers get rated. The older approach leaned on what providers said about themselves: briefings, submitted questionnaires, case studies the provider picked. More of the newer ratings start from the other end, with the companies actually paying for the work and living with the result. That’s a real improvement. It never made much sense for the people being rated to write the rating.

A rating like that answers a hard question well: across a market, which providers their clients tend to be happy with, and on what. If you are about to choose one, that is worth knowing.

It answers a different question from the one most companies get stuck on a year after they sign.

Choosing and Governing Are Two Jobs

Picking a provider is about the market. Once you sign, the question is about your own company: what this purchase has to produce, from what starting point, by when, and what has to be true on your side for it to happen.

No rating can hold that for you. It describes how a provider has done for other companies, averaged and weighted. Your purchase has its own baseline, its own conditions and its own reason for being approved, and those exist only inside your organization. If nobody wrote them down before the contract was signed, there is nothing to check the result against later, however well the provider rates.

Ratings, references and a clean selection all help you choose well. A year later, when someone asks whether the purchase paid off, the selection file tells you why the provider was picked and nothing about what you got.

Ratings Drift Toward Whoever Works Them

Net Promoter Score was meant to be a neutral way for customers to rate the companies they buy from. On the vendor side, I watched it become something to manage. The effort went into finding someone at the customer, sometimes almost anyone, who would say good things, and that person often knew nothing about adoption, results, or the measures the purchase was supposed to move.

Evaluation criteria have the same history. A vendor that reaches the sponsor first can get its own strengths written into how the vendors will be compared, the same way it happens with RFIs and RFPs. Both sides of the table know it happens, and the process keeps running anyway.

A Client’s Rating Is Only as Good as the Client’s Record

Ratings that start with clients are a step in the right direction, and they still depend on what each client actually knows about its own result. If a client wrote down what the purchase was for, with a baseline from its own records, it can say with some confidence whether the provider delivered. Plenty of clients never did, and they’re reporting an impression: the relationship felt good, the go-live happened, the invoices looked reasonable. Both answers end up in the same average, and a reader of the rating has no way to tell them apart.

Putting AI on top of the average will not separate them. I have spent decades around business intelligence, data management and research, and the lesson there never changed: incomplete, stale or unreliable inputs produce confident answers that are wrong, and an aggregate of them can be worse than having nothing. A model will summarize those impressions faster and present them more smoothly, which only makes the missing record harder to notice.

The data usually arrives after the claim, too. When I worked at a business intelligence vendor, the promises were made early, and when the buyer finally handed over a good sample of its data, we often struggled to build a business case strong enough to justify the purchase. In the early days of SaaS that kind of pitch had a name: blue-sky selling. The companies that keep a record of each purchase, from their own data and before the money moves, are the ones whose feedback gives a market rating something solid underneath it.

What It Takes on the Buyer’s Side

The record is small. Before a significant purchase is signed, the business owner who will answer for the result writes one page in the company’s own terms: the outcome, the measure, the baseline from your own records, the date, and the conditions on each side. That page goes into the request, so every provider responds to the same definition, and into the contract, so it is still there at renewal.

After that, the latest actual goes in at each check date. When a result falls behind, the record shows whose condition failed. Across all of a company’s significant purchases, those pages add up to something a CFO or a board can read.

What a Provider That Delivers Gets Out of It

A good provider rarely gets a record, kept by the buyer, that backs up what it delivered. A provider measured against a definition the customer wrote, from a baseline the customer supplied, has evidence its competitors cannot easily match, and it has that evidence at renewal, when it counts. It also stops taking the blame for conditions on the buyer’s side that the record shows were never met.

Use Both

Read the ratings when you choose a provider, and use the best of them. Then, before you sign, write down what this purchase has to produce and how you will know, because no rating will ever hold that for your company.

The Request for Value is the one-page form that defines a purchase before the market is asked, and the Purchase Outcome Register is a free workbook that keeps the record across every significant purchase. Both are on the Guides page.

Keep reading: ACT II · Chapter 11: Outcome Credibility Markets