How AI Performs on the Bar Exam Quiz

Answers and explanations appear as you take the quiz.

1. Which Uniform Bar Exam component has 200 multiple-choice questions and counts for half the total score?
   • The Multistate Performance Test (MPT)
   • The Multistate Essay Examination (MEE)
   • The Multistate Bar Examination (MBE)
   • The Multistate Professional Responsibility Examination (MPRE)
2. What simulated Uniform Bar Exam score did OpenAI's GPT-4 technical report cite in 2023?
   • About 298 out of 400
   • About 250 out of 400
   • About 350 out of 400
   • About 198 out of 400
3. According to Eric Martínez's 2024 critique, why was GPT-4's 90th-percentile figure misleading?
   • The model was shown the answer key before testing
   • It was compared with February test takers, a group that includes many repeat takers
   • Only the performance test section was used
   • The comparison group was law professors, not applicants
4. Who graded GPT-4's essay answers in the 2023 simulated bar exam, according to the critique described in the guide?
   • Official NCBE-trained bar graders
   • A committee of a state bar
   • The researchers themselves, using published sample answers
   • A second AI model working as an automatic grader
5. What did Bommarito and Katz find when they tested GPT-3.5 on MBE practice questions in late 2022?
   • It answered every question correctly
   • It refused to answer legal questions
   • It scored below random chance
   • It answered about half correctly: above chance, below passing
6. Why is training-data contamination a specific concern for AI bar exam results?
   • Past bar questions circulate online and may have been in the model's training data
   • Bar exams are only given in printed form, so models cannot read them
   • The NCBE deliberately publishes wrong answers
   • Models are forbidden from seeing legal text during training
7. Why do researchers simulating a bar exam score have to approximate the conversion from raw to scaled scores?
   • Scaled scores are chosen at random each year
   • Models produce answers that cannot be scored
   • Each jurisdiction uses a completely different set of questions
   • The NCBE's full equating process is not public
8. What does the guide say the NCBE's NextGen bar exam gives more weight to?
   • Memorizing a longer list of rules
   • Integrated skills such as research and drafting
   • Speed on multiple-choice questions
   • Oral argument before a panel of judges