AI Vendor Assessment Checklist: 15 Questions Before You Buy
Before you buy any AI tool, you need to ask the right questions. An AI vendor assessment is a structured evaluation of a tool’s data handling, security, compliance, reliability, and contract terms. It protects your business from vendors who look good in a demo but create problems in production.
Most businesses skip this step. They see a slick product, sign up for a trial, paste in some data, and only start asking questions when something goes wrong. By then, your customer data might already be training someone else’s model, your team might be dependent on a tool with no data export option, and you might be non-compliant with the Privacy Act 2026.
This checklist gives you 15 questions to ask before you commit to any AI vendor. They are organised into five categories: data handling, security, compliance, reliability, and contract terms. Use them every time.
Data Handling (Questions 1 to 4)
Data is the first thing to assess. Where does your data go? Who can access it? Is it being used for purposes you have not agreed to? These four questions cover the essentials.
Where is our data stored and processed?
Data residency matters for compliance. Know whether your data stays in Australia, goes to the US, EU, or elsewhere. Some industries require Australian data residency.
Does the vendor use our data to train their AI models?
Many AI vendors use customer data for model improvement. Enterprise tiers often offer opt-out. If you cannot opt out, your proprietary data may end up improving a model your competitors also use.
Is there a data processing agreement (DPA) available?
A DPA is a legal document that defines how the vendor handles your data. Under the Privacy Act 2026, you need contractual safeguards for any third party processing personal information on your behalf.
How long does the vendor retain our data, and can we request deletion?
Some vendors retain conversation logs, inputs, and outputs for months. You need to know the retention period and whether you can request full deletion of your data.
Security (Questions 5 to 7)
A tool that handles your data needs to protect it. These questions establish whether the vendor takes security seriously or is just ticking boxes.
What security certifications does the vendor hold?
SOC 2 Type II and ISO 27001 are the baseline. ISO 42001 is a strong signal for AI-specific governance. Ask for current certificates, not just claims on a website.
How is data encrypted in transit and at rest?
TLS 1.2 or higher for data in transit and AES-256 for data at rest are industry standards. If the vendor cannot confirm both, that is a red flag.
What access controls exist for our account and data?
Role-based access, multi-factor authentication, SSO integration, and audit logs are all important. The more sensitive your data, the stricter the access controls need to be.
Compliance (Questions 8 to 10)
Australian regulations are tightening. These questions check whether the vendor can support your compliance obligations, particularly around the Privacy Act 2026.
Does the vendor support automated decision-making transparency?
The Privacy Act 2026 requires disclosure when AI makes or influences decisions about individuals. Your vendor needs to support explainability or provide the information you need to make your own disclosures.
Can the vendor provide an AI impact assessment for their product?
A responsible vendor should be able to explain the potential impacts of their AI system, including bias risks, accuracy limitations, and failure modes. If they cannot, they have not done the work.
Does the vendor comply with Australian privacy requirements?
International vendors may not be aware of Australian-specific requirements. Ask whether they have reviewed the Privacy Act, the OAIC guidance, and any sector-specific regulations that apply to your industry.
Reliability (Questions 11 to 13)
A great product is worthless if it goes down when you need it or the vendor disappears. These questions assess operational reliability.
What is the vendor's uptime guarantee and incident history?
Check their status page for the last 12 months. Look for the SLA in their terms. A 99.9% uptime guarantee means nearly 9 hours of potential downtime per year. Know what that means for your operations.
What happens to our data and access if the vendor shuts down or is acquired?
AI startups fail. Large companies acquire and sunset products. Your contract should include data portability rights and adequate notice periods for service discontinuation.
What does the support model look like?
When something breaks, can you reach a human? What is the response time? Is support available in Australian business hours? Free-tier support is often limited to email with multi-day response times.
Contract Terms (Questions 14 to 15)
The contract is where promises become obligations. These two questions address the most common traps in AI vendor agreements.
What are the termination terms and data export options?
Vendor lock-in is real. Know how to leave before you sign up. Check for early termination fees, data export formats, and the timeline for receiving your data after cancellation.
Who is liable if the AI tool causes harm or provides incorrect outputs?
Most AI vendors disclaim liability for output accuracy. That means if their tool hallucinates and your business acts on it, the liability sits with you. Understand this before you rely on the tool for anything consequential.
How to Use This Checklist
Run through these 15 questions every time someone on your team wants to adopt a new AI tool. This includes free tools. Free does not mean risk-free. In fact, free-tier AI tools often have the weakest data protections because you are paying with your data rather than money.
Keep a record of every assessment in a shared location. When a vendor changes their terms (and they will), you have a baseline to compare against. When a new team member asks “can I use this tool?” you have a documented decision rather than a guess.
Not every question will be a dealbreaker for every tool. A brainstorming tool that never touches customer data has different requirements than a tool that processes financial records. Use the checklist proportionally. A quick assessment for low-risk tools. A thorough evaluation for anything touching sensitive data or making decisions about people.
This checklist should be part of your broader AI policy. Include it as an appendix or reference it in your approved tools process. The goal is to make vendor assessment a habit, not a one-off exercise.
Red Flags to Watch For
- Vendor cannot tell you where your data is stored or processed
- No data processing agreement available, or they have never heard of one
- Terms of service allow unrestricted use of your data for model training
- No security certifications (SOC 2, ISO 27001) and no timeline to obtain them
- No data export functionality or the export is in a proprietary format
- Vendor disclaims all liability for output accuracy with no option to negotiate
- No incident response or breach notification process documented
- Unable to explain how their AI model makes decisions (a compliance risk under the Privacy Act)
Need help evaluating AI tools for your business?
Our AI Readiness Review includes a full audit of your current AI tools and vendors, with risk-rated recommendations and a clear action plan.
Get your AI Readiness ReviewFrequently Asked Questions
What should I check before buying an AI tool?
Before buying any AI tool, assess five areas: data handling (where your data goes and whether it is used for training), security (encryption, access controls, certifications), compliance (Privacy Act alignment, data residency, automated decision-making transparency), vendor reliability (uptime, support, financial stability), and contract terms (data portability, termination rights, liability). Use a structured checklist to ensure nothing is missed.
Should AI vendors store data in Australia?
It depends on your industry and the type of data. The Privacy Act requires that overseas data transfers have adequate protections. For many businesses, using vendors that process data in the US or EU is acceptable if proper data processing agreements are in place. However, some industries (particularly government and healthcare) have stricter data sovereignty requirements. If your data includes health records or financial information, Australian data residency is strongly preferred.
How do I know if an AI vendor uses my data for training?
Check three places: the vendor's terms of service, their privacy policy, and their data processing agreement. Look specifically for language about model training, service improvement, or aggregate data use. Many vendors have different policies for consumer and enterprise tiers. The enterprise or business tier often includes opt-out provisions for model training. If the terms are unclear, ask the vendor directly and get the answer in writing.
What certifications should an AI vendor have?
At minimum, look for SOC 2 Type II (security controls) and ISO 27001 (information security management). ISO 42001 (AI management systems) is a newer but increasingly important certification. For vendors handling payment data, PCI DSS compliance is required. For healthcare data, look for HIPAA compliance or equivalent Australian standards. The certifications that matter most depend on your industry and the sensitivity of the data involved.
How often should we reassess our AI vendors?
Review your AI vendors at least annually, with a lighter quarterly check on terms of service changes and any security incidents. You should also trigger a reassessment whenever a vendor announces a major product change, changes their data handling terms, experiences a security breach, or is acquired by another company. Set up Google Alerts for your key AI vendors to catch news that might trigger a review.