Questions to Ask Before Buying AI Software
The 18 questions every buyer should ask any AI vendor before signing, plus how to interpret the answers.
Anyone evaluating an AI vendor, especially first-time AI buyers and procurement teams.
The 6-step process
Follow the steps in order. Skipping is how engagements go sideways.
- 1
Ask about the model: 4 questions
Which AI model powers this? Is it a wrapper on a foundation model (which one)? What happens when that model is deprecated? Do you train your own model on customer data?
- 2
Ask about data handling: 4 questions
Where is data stored geographically? Do you train on customer prompts? How long is data retained? What is the deletion process if we leave?
- 3
Ask about evaluation and quality: 3 questions
How do you measure model accuracy? What is your hallucination rate? Can I see evaluation results on use cases like mine?
- 4
Ask about residency and compliance: 3 questions
Can data be kept in Canada? Are you PIPEDA-aligned? Will you sign a Canadian-law-governed DPA?
- 5
Ask about pricing and lock-in: 2 questions
What does total cost look like at 2x and 5x current usage? What are the exit terms (data export, contract cancellation)?
- 6
Ask about support and references: 2 questions
What support do we get post-launch? Can I talk to 2-3 customers similar to us?
Frequently Asked Questions
What is the single most revealing question to ask an AI vendor?
"Do you train your models on our prompts and uploaded files, and how do we opt out?" The answer tells you about data handling, contract flexibility, and the vendor's default posture all at once. Vendors who are evasive on this one are evasive on the rest.
How do I know if a vendor is being honest about hallucination rates?
Ask for evaluation methodology in writing, not just a number. Real vendors describe how they test (sample size, evaluation criteria, who graded). Vendors who give you a magic number with no methodology are pitching, not informing.
Are there standard contract terms for AI vendors yet?
No, the AI vendor contract landscape is still maturing. Generic SaaS contracts miss key AI-specific clauses (training rights, model version, evaluation methodology, exit data). Always have a lawyer review, ideally one who has done AI vendor contracts before.
What if a vendor will not answer some of these questions?
Walk away. There are enough credible AI vendors that you do not need to settle for one who will not explain their model, data handling, or contract terms. "Trust us" is not an answer.
Can I get a checklist version of this?
Yes. Download the free AI Vendor RFP Template linked below, or use the AI Vendor Selection Advisory service which runs this evaluation for you across a shortlist.
When you want hands-on help
AI Vendor Selection Advisory
Learn more →AI Procurement Support
Learn more →PIPEDA-Ready AI Rollout
Learn more →Free tools that help
Related reading
Need help running this in your business?
Book a 30-minute discovery call. We will tell you honestly whether a consulting engagement is the right next step or whether the free tools are enough.