What happened
The Guardian reports that Ravinder Dhillon, a senior crown prosecutor for the Crown Prosecution Service (CPS), cited a fictitious legal case, CPS – R v J [2024] EWCA Crim 3529, during high court proceedings. The case involved a challenge to the decision to prosecute an 11-year-old boy for the alleged rape of his seven-year-old sister. Dhillon admitted in October that the citation was an error, stating she had intended to reference a real decision from 15 years earlier. The Guardian understands the fictional case details originated from an AI summary generated during an internet search. Two senior high court judges described the CPS's explanation as "unsatisfactory," while legal expert Brian Flanagan called it a "serious failure." The high court ruled that the error did not affect the outcome of the boy's challenge, and the prosecution is set to proceed to trial in youth courts. This incident marks the second known case involving the CPS where AI-related errors led to the citation of nonexistent legal authorities, following a similar admission in February regarding an extradition case.
The Guardian reports that Ravinder Dhillon, a senior crown prosecutor in the CPS’s rape and serious sexual offences unit, cited a nonexistent case, CPS – R v J [2024] EWCA Crim 3529, during high court proceedings. The case involved a legal challenge by the defense of an 11-year-old boy, who was accused of raping his seven-year-old sister, against the CPS’s decision to prosecute him. The defense argued that prosecutors had failed to consider representations from social workers, youth workers, and a forensic psychologist.
Dhillon admitted in October that the citation was an error, explaining that she had intended to reference a real decision made 15 years earlier. The Guardian understands that the fictional case details came from an AI response generated during an internet search. Two senior high court judges described the CPS’s explanation for the error as "unsatisfactory." Brian Flanagan, an associate professor of law at Maynooth University, called the incident a "serious failure," noting that it calls into question norms of trust in the law.
The high court determined that the reference to the fictitious case did not affect the outcome of the boy’s challenge to the decision to continue with his prosecution. The case is listed to go to trial in the youth courts later this week. This is the second known instance where the CPS has admitted to citing nonexistent legal authorities likely caused by AI, following a similar error in February involving an extradition case.
The CPS stated that it has a strict AI policy which limits its use so that it is not used to make legal decisions and requires human checks for accuracy. A spokesperson said the error was identified from conducting legal research using the internet and was remedied promptly. An internal review was carried out to establish how the error occurred and to check that no other cases were affected. The CPS shared learnings, including that case law must be cited from trusted legal sources.
Source details: theguardian.com
Why it matters
This incident highlights the tangible risks of AI hallucinations within the justice system, particularly when state actors rely on unverified AI-generated legal research. The use of a fictitious precedent to support the prosecution of a child undermines public trust in the integrity of legal proceedings and the reliability of legal authorities. It demonstrates that even with internal policies limiting AI use for decision-making, the integration of AI into legal research workflows introduces significant risks of error that can have severe consequences for individuals. The case underscores the need for robust verification protocols and raises questions about accountability when AI tools are used in sensitive legal contexts. It also contributes to a growing body of evidence, tracked by legal professionals, showing that AI-related errors are becoming a recurring issue in court proceedings, potentially affecting the administration of justice and public confidence.
The incident underscores the practical risks of AI hallucinations in the legal system, where the citation of nonexistent precedents can have serious consequences for individuals, particularly in sensitive cases involving minors. It highlights the potential for AI-generated errors to undermine the integrity of legal proceedings and public confidence in the justice system.
The case adds to a growing number of AI-related errors in court proceedings, with a UK tracker maintained by Matthew Lee of Doughty Street Chambers now listing 101 confirmed, suspected, or disputed AI-related errors. This trend suggests that AI misuse is becoming a significant issue in the legal profession, requiring urgent attention from regulatory bodies and professional organizations.
The incident raises questions about the adequacy of current AI policies in legal institutions. While the CPS claims to have strict policies, the occurrence of this error suggests that human checks may not be sufficient to prevent AI-generated misinformation from entering legal submissions. This could lead to calls for more rigorous verification processes and potentially stricter regulations on the use of AI in legal research.
The case also highlights the broader implications of AI integration into professional fields, where the speed and convenience of AI tools may lead to a reduction in careful verification, increasing the risk of errors. This is particularly concerning in fields where accuracy is critical, such as law, medicine, and finance.
Interactive Mechanism: How It Actually Works
Explore the underlying technology behind this development interactively.
Why can ethical evaluation not be reduced to one model score?
What to watch next
Monitor for further disclosures of AI-related errors in legal proceedings, particularly involving public agencies like the CPS. Watch for regulatory or professional body responses, such as new guidelines from the Bar Council or Law Society, aimed at curbing the misuse of AI in legal research. Observe whether the CPS implements additional safeguards or changes its internal AI policy in response to this incident. Track the outcome of the youth court trial to see if the initial error has any lingering impact on the case's proceedings or public perception.
Watch for further reports of AI-related errors in legal proceedings, particularly in high-profile cases or those involving public agencies. The number of such incidents may continue to rise as AI tools become more widely used in legal research.
Monitor responses from legal professional bodies, such as the Bar Council and the Law Society, which have been called upon to consider steps to curb the problem of AI hallucinations in legal submissions. New guidelines or sanctions for misuse of AI may be introduced.
Observe whether the CPS or other legal institutions implement additional safeguards or changes to their AI policies in response to this incident. This could include mandatory verification of all AI-generated legal or restrictions on the use of AI in certain types of cases.
Track the outcome of the youth court trial to see if the initial error has any impact on the case's proceedings or public perception. The case may serve as a test case for how courts handle AI-related errors in legal submissions.