Hallucination Rates in Legal AI Tools Quiz

Answers and explanations appear as you take the quiz.

1. How did the Stanford "Hallucination-Free?" study define a hallucination?
   • Any answer longer than the source document
   • An answer that is either incorrect or misgrounded
   • Only a citation to a case that does not exist
   • Any answer the tool took more than a minute to produce
2. In the Stanford study's terms, what is a 'misgrounded' answer?
   • An answer that refuses to respond
   • An answer written in the wrong language
   • An answer with no citations at all
   • An answer that cites a real source that does not support its claim
3. Which tool performed best in the 2024 Stanford test but still hallucinated on more than 17 percent of queries?
   • Lexis+ AI
   • Westlaw AI-Assisted Research
   • Ask Practical Law AI
   • GPT-4 used as a comparison
4. What hallucination range did "Large Legal Fictions" find for general-purpose chatbots on verifiable federal court questions?
   • 1 to 3 percent
   • 17 to 33 percent
   • 69 to 88 percent
   • 95 to 100 percent
5. Why can a retrieval step find the wrong authority even when the database contains the right one?
   • Legal databases are encrypted and cannot be searched
   • Retrieval only works for statutes, not cases
   • Retrieval ranks documents alphabetically
   • Retrieval matches similar wording or meaning, which is not the same as legal relevance
6. What tendency did the research find when questions contained a false premise?
   • Models tended to accept the premise and answer as if it were true
   • Models always corrected the premise
   • Models refused every such question
   • Models replied only with the text of the relevant statute
7. Which mitigation specifically helps flag a cited case that was later reversed or overruled?
   • Raising the model's temperature setting
   • Making answers shorter
   • Translating the query into another language
   • Connecting a citator such as KeyCite or Shepard's
8. What marketing claim did the Stanford study's title directly question?
   • That legal AI tools were cheaper than paralegals
   • That some tools were 'hallucination-free'
   • That legal AI could pass the bar exam
   • That AI tools were approved by courts