言語AIガイド

How AI Performs on the Bar Exam

Large language models such as GPT-4 have scored above typical passing lines on simulated versions of the Uniform Bar Exam.

4 分で読めます
最終更新日 2026年9月15日

読み始めるクイズに答えてください

このページでは4 分で読めます

概要
ディープダイブ
戦略的影響
The Future of How AI Performs on the Bar Exam
現実世界の実装
リスクとガードレール
実装ロードマップ
探検を続けましょう
よくある質問

概要

The best-known figure is about 298 out of 400, reported in 2023. These results show that models can handle the written, rules-heavy format of a licensing test. They measure test performance under particular grading and comparison choices, not the full judgment a practicing lawyer needs.

ディープダイブ

The Uniform Bar Exam (UBE) has three parts. The Multistate Bar Examination (MBE) has 200 multiple-choice questions and counts for half the score. The Multistate Essay Examination (MEE) has six essays. The Multistate Performance Test (MPT) has two tasks in which the examinee works from a closed file of facts and law to write a memo or brief. Scores run to 400, and each jurisdiction sets its own passing line, commonly between 260 and 270. In late 2022, Michael Bommarito and Daniel Martin Katz tested GPT-3.5 on MBE practice questions. It answered about half correctly: well above chance, but below passing. In March 2023, Katz, Bommarito, Shang Gao and Pablo Arredondo reported that GPT-4 scored about 297 on a full simulated UBE. OpenAI's GPT-4 technical report cited roughly 298 and a percentile near the 90th. The percentile drew the sharpest criticism. In a 2024 paper in the journal Artificial Intelligence and Law, Eric Martínez showed that the 90th-percentile figure was based on February test takers. The February group includes an unusually large share of people retaking the exam after failing. Compared with first-time July takers, his estimate fell to around the 60th percentile, and lower on the essays. He also noted that the essays were scored by the researchers against published sample answers, not by trained bar graders, which adds uncertainty. Contamination is another concern: past questions circulate online and may have been in the training data. A common misconception is that passing the bar means a model can practice law. The exam tests recall of rules and structured analysis within a closed set of facts. Practice also requires investigating facts, counseling clients, planning strategy, judging risk and taking professional responsibility. In Mata v. Avianca (2023), lawyers filed a brief containing citations that ChatGPT had invented. The case showed that exam-level fluency does not guarantee reliable legal research.

技術的な洞察

Reproducing a bar score involves choices that change the result. Researchers pick a prompting format: zero-shot, few-shot, or asking the model to reason step by step before answering. For MBE items, they then extract a single answer letter from the output. Raw MBE results have to be converted to a scaled score. The NCBE's equating process is not published in full, so studies approximate it from available conversion data. In real exams, written scores are scaled against the MBE, and a simulation has to imitate that step. Percentiles depend entirely on which group of test takers is used for comparison. Contamination checks look for overlap between test items and training text, but closed training sets make this hard to verify. When evaluating legal models, practitioners should prefer newly written questions the model cannot have seen. They should report who graded and with what rubric, run multiple samples to measure variance, and score the reasoning, not only the final answer.

戦略的影響

速度とスケール

言語ワークフローは、一貫性を犠牲にすることなく、より高速に移行できます。

アクセスと到達範囲

言語やコミュニケーションスタイルを超えてアクセスが拡張されます。

より明確な判決

自動化が繰り返しを処理する間、チームは判断により多くの時間を費やすことができます。

The Future of How AI Performs on the Bar Exam

Bar scores are becoming less useful as a benchmark. Top models already score near the ceiling on multiple-choice legal questions, and contamination is hard to rule out. Researchers have moved toward task collections such as LegalBench, a collaborative benchmark released in 2023 that breaks legal reasoning into many narrower tasks. They have also moved toward tests built on real work, such as research memos and contract review, graded by practicing lawyers. The NCBE's NextGen bar exam, planned to begin in 2026, gives more weight to integrated skills such as research and drafting. That may make comparisons with AI more informative. Whether models can pass tests is largely settled. The open question is how reliably they perform, and how they fail, on unfamiliar real matters.

現実世界の実装

OpenAI's March 2023 GPT-4 technical report listed a simulated Uniform Bar Exam score of about 298 out of 400 and a percentile near the 90th, a figure that was then widely repeated in headlines.

A law professor runs past MBE-style multiple-choice questions through a chatbot and compares its answers with her students'. The model is strong on black-letter rules but less reliable when the facts are deliberately ambiguous.

An evaluator checks whether a practice question was posted online before a model's training cutoff. A model that has already seen the question may be recalling an answer rather than reasoning to it.

A bar applicant uses an AI tutor to explain the MBE questions she missed. She checks each explanation against a commercial outline because the tutor sometimes presents a minority rule as the majority rule.

リスクとガードレール

幻覚のような事実が、レポート、サポートフロー、または研究結果に静かに組み込まれる可能性があります。
迅速な対応により、同様のリクエスト間で一貫性のない結果が生じる可能性があります。
アクセス制御が弱いと、機密テキストデータが漏洩する可能性があります。

実装ロードマップ

展開する前に、出力形式、トーン、品質基準を定義します。
正確さが重要な場合は常に、信頼できる情報源を使って地上対応を行ってください。
一か八かの成果物については人間によるレビューチェックポイントを維持します。
失敗パターンを追跡し、プロンプトやワークフローを定期的に再トレーニングします。

探検を続けましょう

ChatGPTとLLM

NLPの基礎

これらのツールの背後にある言語処理の基礎を学びます。ガイドを読む

Free newsletter

Three verified AI stories every weekday morning, written in plain English. Free forever, no ads.

One email each weekday. Unsubscribe in one click. We never sell or share your address.

Test yourself

Take the How AI Performs on the Bar Exam quiz

Instant feedback on every answer, and a shareable certificate with a verifiable ID once you pass a course.

クイズを開始する

Support free AI education. AI Understanding is a 501(c)(3) nonprofit — no ads, no paywall, ever. Make a donation

よくある質問

What is How AI Performs on the Bar Exam?

Large language models such as GPT-4 have scored above typical passing lines on simulated versions of the Uniform Bar Exam. The best-known figure is about 298 out of 400, reported in 2023. These results show that models can handle the written, rules-heavy format of a licensing test. They measure test performance under particular grading and comparison choices, not the full judgment a practicing lawyer needs.

Which Uniform Bar Exam component has 200 multiple-choice questions and counts for half the total score?

The MBE is the 200-question multiple-choice section and makes up 50 percent of the UBE score. The MEE is six essays, and the MPT is two closed-file tasks.

What simulated Uniform Bar Exam score did OpenAI's GPT-4 technical report cite in 2023?

OpenAI reported roughly 298 out of 400. That is above the 260 to 270 passing range common across jurisdictions, and it was paired with a percentile near the 90th.

According to Eric Martínez's 2024 critique, why was GPT-4's 90th-percentile figure misleading?

The February group includes a large share of people retaking the exam after failing. Compared with first-time July takers, Martínez estimated GPT-4 fell to around the 60th percentile overall.

Who graded GPT-4's essay answers in the 2023 simulated bar exam, according to the critique described in the guide?

The essays were scored by the study's authors against published sample answers, not by trained bar graders. Martínez noted this adds uncertainty to the written-section scores.

What did Bommarito and Katz find when they tested GPT-3.5 on MBE practice questions in late 2022?

GPT-3.5 got roughly half the questions right. That is well above the 25 percent expected from guessing on four-option questions, but below a passing level.

学び続ける