How to Evaluate an AI Vendor
A 12-point checklist for evaluating any AI vendor, from foundation models to vertical SaaS, before you sign.
Procurement, IT, legal, and ops leaders evaluating any AI vendor before contract.
The 7-step process
Follow the steps in order. Skipping is how engagements go sideways.
- 1
Confirm what AI model is actually under the hood
Ask which model powers the product, whether it is a wrapper on a foundation model (and which one), and what happens when the underlying model is updated or deprecated.
- 2
Verify how your data is handled
Does the vendor train on your prompts by default? Can you opt out? Where is data stored geographically? How long is it retained? Get answers in writing, in the DPA.
- 3
Test for hallucination on your actual data
Bring a sample of your real-world inputs and run them. Ask follow-up questions where the right answer is "I do not know." A good AI vendor handles uncertainty; a bad one confabulates.
- 4
Score the vendor across 12 dimensions
Use a weighted matrix covering: model quality, accuracy, evaluation methodology, data handling, residency, integration depth, support quality, pricing model, exit terms, references, security posture, and AIDA/PIPEDA alignment.
- 5
Demand real references
Ask for 2-3 customers similar to you in size and industry. Talk to them about what went wrong, not just what went right. "Nothing went wrong" is a red flag.
- 6
Negotiate the contract on AI-specific terms
Pricing escalators (especially per-token or per-seat-spike), residency commitments, training-data rights, model-version control, exit data export, and incident notification windows.
- 7
Plan the exit before you sign the entry
What happens to your data if the vendor disappears, raises prices 3x, or is acquired by a competitor? If you do not have a clear answer, you are not ready to sign.
Frequently Asked Questions
What is the single biggest red flag in an AI vendor pitch?
When the vendor cannot or will not tell you which underlying model they use, where data is stored, or whether they train on prompts. That is not "confidential," that is "we hope you do not ask."
Do AI vendors really train on customer prompts?
Many do, by default. Some opt-out is buried in account settings or contract terms. Always verify in writing whether your prompts and uploaded files are used for training, and get an opt-out commitment in the DPA, not just the marketing page.
What is the difference between SOC 2 and PIPEDA compliance for AI vendors?
SOC 2 covers security controls. PIPEDA covers personal-information privacy. An AI vendor can be SOC 2 compliant and still mishandle Canadian personal information under PIPEDA. Both matter, but they answer different questions.
How long should AI vendor evaluation take?
For a $5K-$20K annual purchase, 2-4 weeks is right. For $50K+, 4-8 weeks. Faster than that and you are skipping steps; slower and you are over-investing for the deal size.
Should I use a consultant for vendor evaluation?
For one tool under $10K, usually no. For multi-vendor evaluations, enterprise procurement, or anything in a regulated industry, yes. AI Vendor Selection Advisory exists for exactly this.
When you want hands-on help
Free tools that help
Related reading
Need help running this in your business?
Book a 30-minute discovery call. We will tell you honestly whether a consulting engagement is the right next step or whether the free tools are enough.