AI Vendor Evaluation Checklist Quiz

Answers and explanations appear as you take the quiz.

1. What is the key difference between a SOC 2 Type II report and a Type I report?
   • Type II covers only physical security
   • Type I is newer and more thorough
   • Type II covers how controls operated over a period, while Type I describes them at a single point in time
   • Type I applies only to AI vendors
2. According to the guide, what is the strongest accuracy evidence a vendor can provide?
   • A pilot on a test set you build from real cases, scored by your own staff
   • A polished live demo
   • Published marketing case studies
   • A claim that the product uses the most advanced model available
3. In AI Vendor Evaluation Checklist: what is ISO/IEC 42001?
   • A data privacy law in the European Union
   • A standard for AI management systems
   • A cloud pricing benchmark
   • A penetration testing tool
4. Where should a vendor's answers about data retention and training on customer data be documented?
   • In a sales email
   • In a slide deck from the demo
   • In a verbal assurance during a call
   • In the contract or data processing agreement
5. Which of the following is a red flag in an AI vendor evaluation?
   • The vendor offers to run a pilot on your data
   • The vendor claims 100 percent accuracy
   • The vendor names its subprocessors
   • The vendor provides a SOC 2 Type II report
6. Why should you keep a held-out portion of your pilot test set that vendors never receive?
   • To reduce the cost of the pilot
   • Because vendors are legally barred from seeing test data
   • So vendors cannot tune their product to the specific test cases
   • To make the pilot run faster
7. Why does the guide recommend recording the model version and configuration used during testing?
   • A later change to the upstream model can invalidate the pilot results
   • It is required to get a SOC 2 report
   • It reduces the price of the contract
   • It prevents prompt injection attacks
8. Why measure types of errors instead of relying on a single accuracy number?
   • Accuracy numbers are always inflated by exactly 10 percent
   • Vendors are not allowed to report accuracy
   • Error types are easier to calculate
   • A rare but severe error can matter more than many minor ones