AI in Courts and Judicial Risk Assessment Quiz

Answers and explanations appear as you take the quiz.

1. What does the Public Safety Assessment (PSA) produce?
   • A single guilt probability for use at trial
   • A recommended prison sentence in months
   • Separate scores for failure to appear and new criminal activity, plus a flag for new violent criminal activity
   • A psychological profile based on a long interview
2. What did ProPublica's 2016 analysis of COMPAS in Broward County report?
   • Black defendants who did not reoffend were nearly twice as likely as white defendants to be labeled higher risk
   • COMPAS scores were identical for all defendants
   • White defendants were more often labeled high risk and then did not reoffend
   • COMPAS predicted reoffending perfectly
3. How did Northpointe respond to ProPublica's findings?
   • It withdrew COMPAS from the market
   • It admitted the tool was intentionally biased
   • It said race was the most important input
   • It argued the scores were equally predictive: a given score matched similar reoffense rates across races
4. Why can a risk score generally not be calibrated and have equal false positive and false negative rates across groups at the same time?
   • Because the models are too small
   • Because the groups have different underlying base rates of the outcome
   • Because judges refuse to use calibrated tools
   • Because ROC curves cannot be computed for criminal data
5. What did the Wisconsin Supreme Court decide in State v. Loomis (2016)?
   • COMPAS scores are banned in all sentencing
   • Defendants must be shown COMPAS source code
   • A judge may consider a COMPAS score with warnings about its limits, but it cannot be the deciding factor
   • Risk scores must replace judicial discretion
6. What is 'label bias' in pretrial risk tools?
   • Rearrest, the outcome being predicted, reflects police contact rather than actual offending
   • Defendants are given inaccurate name labels in the database
   • Scores are printed in colors that bias judges
   • Labels are assigned randomly during training
7. What does the area under the ROC curve (AUC) measure for a risk tool?
   • Whether scores mean the same thing across racial groups
   • How many defendants a court processes per day
   • The legal validity of the tool under due process
   • How well the scores rank people who later fail above those who do not
8. According to the guide, which statement about most judicial risk assessment tools is accurate?
   • They are advanced deep learning systems that read case files
   • They are mostly transparent statistical models that use only a handful of factors
   • They make final bail decisions without a judge
   • They are only used in federal courts