AI Vendor Evaluation Checklist and Scorecard

AI Vendor Evaluation Checklist and Scorecard

To evaluate an AI vendor, score each shortlisted product on four things it can prove rather than four things it can demo: that it stays accurate on your own cases after the model underneath it changes, that it knows when it is unsure and hands those cases to a person, that it can show you why it took any single decision, and that you can leave with your data, configuration and history intact. Features, integrations and price come after those four. The scorecard below turns that into seven weighted rows you can fill in for three or four vendors side by side.

Most vendor checklists in circulation score the opposite way. They weight model performance and feature coverage, which is what a demo is built to win, and say little about the exception path, the audit trail or the exit, which is where a product either survives its second year or gets ripped out.

The Scorecard: Seven Rows, Weighted

Agree the weights before the first demo and score each vendor independently. The weights below lean towards what predicts whether the product is still trusted a year after go-live. Change them if your situation demands it, but change them before you have a favourite.

Score every row from 0 to 5 using the same anchors, so a 3 means the same thing for every vendor:

• 0: not offered.
• 1: claimed in the sales conversation, no evidence.
• 3: shown working, on the vendor's data.
• 5: shown working on your data, and written into the contract.

Multiply each score by its weight and add the rows. The total matters less than the rows that score 0 or 1: a vendor that scores 1 on exceptions or exit is a risk no feature list offsets.

Row 1: Evaluation Suite

An evaluation suite is a set of your own past cases with known-correct answers, run against the product automatically so accuracy becomes a number rather than an impression. It carries the most weight because it is the only row that keeps telling you the truth after you sign. The model inside an AI product is not fixed: model providers retire versions on a published schedule, and Anthropic, for one, commits to at least 60 days' notice before retiring a publicly released model. When the vendor moves to the replacement, your accuracy moves with it, and only an evaluation suite will show you which way.

Ask:

• Can we run our own labelled cases through the product during the trial, and see results per step rather than one overall score?
• Can we rerun the same set later, on our own schedule, without a professional-services ticket?
• Which model and version runs underneath, how much notice do we get before it changes, and can we test the new version before it reaches production?
• Do you report accuracy to us monthly, on our cases, or only at renewal?

Red flag: one accuracy percentage from the vendor's own benchmark dataset, or "the model improves automatically". What keeps a system accurate after launch is covered in is your AI system still right six months later.

Row 2: Exceptions

Every workflow has a happy path that demos well and an exception path that decides whether the product survives month two: the invoice that cites a purchase order nobody can find, the supplier who sends a photo instead of a PDF, the approval that goes to a different manager when the usual one is away. A demo shows the first. Your operations team lives in the second.

Ask:

• What does the product do when its confidence is low? Can we set the threshold ourselves, per step?
• Where do uncertain cases go, and can we route them to a named person with the original document and the product's reasoning attached?
• Can we see the exception queue, how many cases sat in it last week and how they were resolved?
• Can we keep a step permanently with a person, so the product prepares the case and never closes it?

Bring a list of the exceptions your process actually has, ideally from an Exception Ledger, and make each vendor show you what happens to them. Where a person should approve, design for human in the loop on the irreversible steps, not on every step.

Red flag: "the model handles edge cases." That sentence is where pilots die.

Row 3: Decision Log

A decision log, or AI audit trail, records for every automated action what the product saw, what it decided, how confident it was, which model version decided it and who approved it. It is ordinary engineering, not innovation, but without it nobody in finance or compliance can defend a decision the product took, and the product will not be allowed near anything that matters.

For some uses it is also law. Under the EU AI Act, high-risk AI systems must technically allow the automatic recording of events over their lifetime (Article 12), and deployers must keep the logs those systems generate for at least six months unless other law says otherwise (Article 26). Whether your use case is high-risk is a question for your counsel. Whether the product can produce the log is a question for the vendor, and you should ask it either way.

Ask:

• Show us the log entry for one decision, end to end, with the inputs, output, confidence, model version and approver.
• Can compliance read it without an engineer, and can we export it to our own storage?
• How long is it retained by default, and can we set that ourselves?

Red flag: "the model can explain its reasoning." An explanation the model writes after the fact is not a record of what happened.

Row 4: Handover and Exit

Every product you license will one day be renegotiated, replaced or retired. The exit row prices that day in now, while you still have leverage. What you need to walk away with is the work that made the product useful to you: the configuration, the prompts and rules you tuned, the labelled cases you built the evaluation suite from, and the decision log.

Ask:

• In what format can we export configuration, prompts, labelled cases and logs, and can we test an export during the trial?
• Does the processor agreement require you to delete or return all our personal data at the end of the service, at our choice? GDPR Article 28(3)(g) requires that term in any processor contract.
• Can our own team change thresholds, rules and routing without your professional services?
• What termination assistance is included, and for how long?

The same documents matter when the system is built for you rather than bought. What an AI implementation partner should hand over lists them and who reads each one.

Red flag: your tuned configuration only exists inside the vendor's platform, in a format nothing else reads.

Row 5: Fit With the Systems You Keep

A workflow is slow because of its exceptions and handoffs, not because of the database underneath it. An AI product that wants to become your new system of record turns a workflow fix into a migration project. Score this row on whether the product reads from and writes back to the ERP, CRM or gestionale you already run, through their APIs, so nothing moves and nothing is decommissioned. That commitment has a name, the ERP-additive covenant, and it is cheap for a vendor to put in writing if it is true.

Red flag: phase one is a data migration into the vendor's platform.

Row 6: Security and Data Protection

This row is table stakes, which is why it carries 10% rather than 30%: every serious vendor will pass most of it, and passing it tells you nothing about whether the product works. Fail it and the vendor is out, whatever the total.

• A current SOC 2 Type II report, which tests controls over a period, rather than a Type I, which describes them at a point in time. Read the scope: it should cover the product you are buying.
• ISO/IEC 27001 certification, with a scope that includes the service.
• A GDPR processor agreement that lets you, or an auditor you appoint, audit the vendor (Article 28(3)(h)), plus the list of sub-processors, including the model provider.
• Where your data is stored and processed, and a written commitment that it is not used to train models.

Row 7: Commercial Model

Last, and lightest. The question is not which vendor is cheapest today but whether you can predict next year's bill. Pricing that scales per model call grows with your volume, and every step the product routes through a model costs money on every run, which is one reason most of an AI system should not be AI. Ask how price changes when the vendor changes the underlying model, what counts as a billable unit, and what is excluded.

How to Run the Evaluation

1. Write down the workflow, not the features. What arrives, from whom, which systems it touches, and the exceptions you already know about. Vendors who cannot map their product onto that description are telling you something.
2. Shortlist three or four. Include at least one option that is not a new product at all: the AI features in software you already license, or building the workflow instead of buying it.
3. Bring your own cases. Give every vendor the same sample of your past cases, weighted towards the exceptions, with the correct answers held back. This is the start of your evaluation suite, and it stays yours whichever vendor wins.
4. Run in shadow mode before you score. In shadow mode the product works on live cases and records what it would have done while your team still does the work. It is the cheapest way to turn a 3 into a 5 on rows 1 and 2.
5. Score independently, then compare. Operations scores rows 1, 2 and 5. Compliance and IT score rows 3, 4 and 6. Procurement scores row 7. One sponsor breaks ties.
6. Put the fives in the contract. Anything that scored 5 because the vendor promised it belongs in the contract: the evaluation rerun, the model-change notice, the log export, the data return.

Red Flags in an AI Vendor Evaluation

• Accuracy only on the vendor's dataset, never on yours.
• No answer to "what happens when it is unsure?" beyond "it is rarely unsure".
• No per-decision log, or one only engineers can read.
• Exports that work for data but not for configuration, so leaving means rebuilding.
• Migration as phase one.
• A model change you hear about from your users rather than from the vendor.

Sources & References