概述
最著名的數字是 2023 年報告的 400 項中的 298 項。這些結果表明,模型可以處理書面的、規則繁重的許可測試格式。他們衡量特定評分和比較選擇下的測試表現,而不是執業律師所需的全面判斷。
深入探討
The Uniform Bar Exam (UBE) has three parts. The Multistate Bar Examination (MBE) has 200 multiple-choice questions and counts for half the score. The Multistate Essay Examination (MEE) has six essays. The Multistate Performance Test (MPT) has two tasks in which the examinee works from a closed file of facts and law to write a memo or brief. Scores run to 400, and each jurisdiction sets its own passing line, commonly between 260 and 270. In late 2022, Michael Bommarito and Daniel Martin Katz tested GPT-3.5 on MBE practice questions. It answered about half correctly: well above chance, but below passing. In March 2023, Katz, Bommarito, Shang Gao and Pablo Arredondo reported that GPT-4 scored about 297 on a full simulated UBE. OpenAI's GPT-4 technical report cited roughly 298 and a percentile near the 90th. The percentile drew the sharpest criticism. In a 2024 paper in the journal Artificial Intelligence and Law, Eric Martínez showed that the 90th-percentile figure was based on February test takers. The February group includes an unusually large share of people retaking the exam after failing. Compared with first-time July takers, his estimate fell to around the 60th percentile, and lower on the essays. He also noted that the essays were scored by the researchers against published sample answers, not by trained bar graders, which adds uncertainty. Contamination is another concern: past questions circulate online and may have been in the training data. A common misconception is that passing the bar means a model can practice law. The exam tests recall of rules and structured analysis within a closed set of facts. Practice also requires investigating facts, counseling clients, planning strategy, judging risk and taking professional responsibility. In Mata v. Avianca (2023), lawyers filed a brief containing citations that ChatGPT had invented. The case showed that exam-level fluency does not guarantee reliable legal research.
現實世界的實施
OpenAI 2023 年 3 月的 GPT-4 技術報告列出了模擬的統一律師考試分數,滿分 400 分,約為 298 分,百分位接近第 90 位,這一數字隨後在頭條新聞中廣泛重複。
一位法學教授透過聊天機器人運行 MBE 風格的多項選擇題,並將其答案與學生的答案進行比較。這個模型在黑字規則方面很強大,但當事實故意模稜兩可時就不那麼可靠了。
評估人員會檢查練習題是否在模型訓練截止之前在線上發布。已經看到問題的模型可能會回憶答案,而不是對其進行推理。
一位律師申請者使用人工智慧導師來解釋她錯過的 MBE 問題。她根據商業大綱檢查每個解釋,因為導師有時會提出少數規則作為多數規則。
AI 在律師考試表現的未來
條形分數作為基準的用處越來越小。頂級模型在多項選擇法律問題上的得分已經接近上限,很難排除污染的可能性。研究人員已轉向任務集合,例如 LegalBench,這是 2023 年發布的協作基準,它將法律推理分解為許多更狹窄的任務。他們還轉向基於實際工作的測試,例如由執業律師評分的研究備忘錄和合約審查。 NCBE 的 NextGen 律師考試計劃於 2026 年開始,更加重視研究和起草等綜合技能。這可能會讓與人工智慧的比較提供更多資訊。模型能否通過測試已經基本確定。懸而未決的問題是,他們在不熟悉的實際問題上表現如何可靠,以及如何失敗。
付諸實踐
為什麼這很重要
- 速度與規模. 語言工作流程可以在不犧牲一致性的情況下更快地移動。
- 交通與覆蓋範圍. 它擴展了跨語言和溝通方式的訪問。
- 更明確的決策. 團隊可以花更多時間進行判斷,而自動化則可以處理重複。
留意
- 幻覺的事實可以悄悄地進入報告、支持流程或研究成果。
- 及時的敏感性可能會在類似的請求中產生不一致的結果。
- 如果存取控制薄弱,敏感文字資料可能會暴露。
後續步驟
- 在推出之前定義輸出格式、語氣和品質標準。
- 當準確性很重要時,請使用可信任來源進行地面回應。
- 為高風險輸出保留人工審查檢查點。
- 追蹤故障模式並定期重新訓練提示或工作流程。
不斷探索
常見問題
什麼是 AI 在律師考試中的表現?
GPT-4 等大型語言模型在統一律師考試的模擬版本中得分高於典型的及格線。最著名的數字是 2023 年報告的 400 項中的 298 項。這些結果表明,模型可以處理書面的、規則繁重的許可測試格式。他們衡量特定評分和比較選擇下的測試表現,而不是執業律師所需的全面判斷。
統一律師考試哪個部分有 200 個選擇題,佔總分的一半?
MBE 是包含 200 個問題的多項選擇部分,佔 UBE 分數的 50%。 MEE 是六篇論文,MPT 是兩篇閉卷任務。
OpenAI 的 GPT-4 技術報告在 2023 年引用了哪些模擬統一律師考試分數?
OpenAI 報告稱,400 分中約有 298 分。這高於各司法管轄區常見的 260 至 270 分及格範圍,並且與第 90 名附近的百分位數配對。
According to Eric Martínez's 2024 critique, why was GPT-4's 90th-percentile figure misleading?
The February group includes a large share of people retaking the exam after failing. Compared with first-time July takers, Martínez estimated GPT-4 fell to around the 60th percentile overall.
根據指南中描述的評論,誰對 2023 年模擬律師考試中 GPT-4 的作文答案進行了評分?
The essays were scored by the study's authors against published sample answers, not by trained bar graders. Martínez noted this adds uncertainty to the written-section scores.
Bommarito 和 Katz 在 2022 年底在 MBE 練習題上測試 GPT-3.5 時發現了什麼?
GPT-3.5 got roughly half the questions right.這遠高於猜測四選項問題時預期的 25%,但低於及格水平。