引導式測驗 · 中等 等級

LegalBench 和評估人工智慧在法律任務中的測驗

測試 LegalBench 範圍、任務多樣性、指標限制、污染、許可以及如何在不過度宣稱法律能力的情況下解釋模型結果。

問題 1 的 8

What does the LegalBench paper describe?

預覽每個問題8 問題

當您參加測驗時,就會出現答案和解釋。

  1. What does the LegalBench paper describe?
    • A certification that a model can replace a law firm
    • A collaborative benchmark of 162 tasks spanning six types of legal reasoning
    • A benchmark covering only clause extraction in commercial contracts
    • A database that searches all federal and state statutes for citations
  2. A model performs well on a hearsay-classification task. Which conclusion is justified?
    • It can apply current hearsay law in every jurisdiction and factual setting
    • It performed well on that task under the evaluation setup
    • It matches lawyer performance across all benchmark categories
    • It can advise clients on evidence without reviewing the underlying facts
  3. Why should an evaluator report results by task instead of only one aggregate score?
    • A single overall score remains representative even when task types are imbalanced
    • One overall score necessarily predicts success in a legal workflow
    • Task-specific results are unnecessary when a benchmark has many examples
    • Different tasks have different formats and scoring, so an average may hide uneven behavior
  4. What can happen when a benchmark is public and widely used?
    • Public task materials may overlap with examples a model encountered during training
    • Contamination occurs only when task answers are used in supervised fine-tuning
    • Expert-authored tasks cannot have appeared in pretraining data
    • Public benchmark exposure does not affect performance if prompts are hidden
  5. Which details make a LegalBench result reproducible?
    • Task subset, prompt, model version, scoring method, and data provenance
    • The data source and task list, but no prompt or scoring instructions
    • The model version and runtime, but not task subset or answer normalization
    • The legal topic and the highest score from a previous evaluation
  6. Before comparing two systems on LegalBench, what should the evaluator check?
    • Same task data with different answer normalization across systems
    • Same task, prompt, scoring, and evaluation data for both systems
    • Same model API but different prompts and decoding settings
    • Same benchmark name while allowing each team to select its own test subset
  7. Why inspect task-specific instructions and licenses in the LegalBench repository?
    • Every task is public domain because it appears in a repository
    • The repository’s top-level license applies to all third-party datasets without exception
    • Tasks draw on different source datasets, and their use terms may differ
    • A task’s instructions establish the current law in every jurisdiction
  8. A model scores well on public legal reasoning tasks. Which next step tests whether it fits a real workflow?
    • Repeat evaluation on another public split and treat it as representative of the target workflow
    • Tune prompts on the held-out examples, then report the same set as independent evaluation
    • Test only tasks that previously failed, then report the result as overall legal competence
    • Test on held-out examples that resemble the intended workflow and have legal experts inspect errors