Vendor evaluation

AI vendor selection fails differently than normal software selection: the product's judgment quality on your data is the product, and it cannot be read off a feature list. The discipline is a scored rubric plus a real proof-of-concept — and writing the POC's success criteria before the first sales call, while your skepticism is still intact.

The six dimensions (score 1–5, weight per initiative)

• Capability fit on YOUR data — not the demo's. The POC (below) is this dimension's only valid evidence.
• Data handling — where does your data go, is it retained, is it used for training, where is it processed, can you delete it? Get answers in the contract, not the slide deck. Your governance charter (Module 4) will set the floor here; anything below the floor is disqualified regardless of capability.
• Security posture — SSO (single sign-on, so staff use one governed corporate login) and access controls, audit logs, certifications appropriate to your sector, breach history and disclosure behavior. Loop in whoever owns security at shortlist time, not at signature time — retrofitted security review is the classic two-month procurement stall.
• Lock-in & exit — can you export your data, configurations, and history? What breaks if you leave? Prefer vendors whose value is ongoing capability over vendors whose value is accumulated hostage data. Ask the exit question directly and watch how they answer — evasion here predicts evasion everywhere.
• Pricing model vs. your volume curve — per-seat, per-use, per-outcome? Model the bill at 1×, 3×, and 10× your pilot volume; per-use pricing that's charming at pilot scale can be punitive at rollout. Negotiate the scale tier now, while you're small and they're eager.
• Vendor viability & velocity — funding runway or profitability, release cadence, roadmap credibility, and what happens to you in an acquisition. The AI vendor landscape is consolidating; you're picking a multi-year dependency during a shake-out. Boring stable beats exciting fragile for anything load-bearing.

The POC that predicts (instead of impresses)

1. 1Write success criteria first: 'extraction error <5% on our 200-document sample, including the 40 ugly ones' — numbers you'd defend, on data you chose, with the worst inputs deliberately included.
2. 2Use your real mess: your formats, your typos, your scanned-sideways PDFs, your angriest exception emails. Vendors who resist testing on your data are answering the capability question for you.
3. 3Run your people, not their engineers: the POC should predict your Tuesday, so your dispatcher configures and your CS lead reviews outputs, with vendor support at arm's length.
4. 4Time-box and score: two to four weeks, scored against the pre-written criteria in front of the steering group. A POC that 'went well' without numbers is a vibe; you're buying with other people's money.