Guide-linked quiz · Medium Level

LegalBench and Evaluating AI on Legal Tasks Quiz

Tests LegalBench scope, task variety, metric limits, contamination, licensing, and how to interpret model results without overclaiming legal competence.

Question 1 of 8

What does the LegalBench paper describe?

Preview every question8 questions

Answers and explanations appear as you take the quiz.

  1. What does the LegalBench paper describe?
    • A certification that a model can replace a law firm
    • A collaborative benchmark of 162 tasks spanning six types of legal reasoning
    • A benchmark covering only clause extraction in commercial contracts
    • A database that searches all federal and state statutes for citations
  2. A model performs well on a hearsay-classification task. Which conclusion is justified?
    • It can apply current hearsay law in every jurisdiction and factual setting
    • It performed well on that task under the evaluation setup
    • It matches lawyer performance across all benchmark categories
    • It can advise clients on evidence without reviewing the underlying facts
  3. Why should an evaluator report results by task instead of only one aggregate score?
    • A single overall score remains representative even when task types are imbalanced
    • One overall score necessarily predicts success in a legal workflow
    • Task-specific results are unnecessary when a benchmark has many examples
    • Different tasks have different formats and scoring, so an average may hide uneven behavior
  4. What can happen when a benchmark is public and widely used?
    • Public task materials may overlap with examples a model encountered during training
    • Contamination occurs only when task answers are used in supervised fine-tuning
    • Expert-authored tasks cannot have appeared in pretraining data
    • Public benchmark exposure does not affect performance if prompts are hidden
  5. Which details make a LegalBench result reproducible?
    • Task subset, prompt, model version, scoring method, and data provenance
    • The data source and task list, but no prompt or scoring instructions
    • The model version and runtime, but not task subset or answer normalization
    • The legal topic and the highest score from a previous evaluation
  6. Before comparing two systems on LegalBench, what should the evaluator check?
    • Same task data with different answer normalization across systems
    • Same task, prompt, scoring, and evaluation data for both systems
    • Same model API but different prompts and decoding settings
    • Same benchmark name while allowing each team to select its own test subset
  7. Why inspect task-specific instructions and licenses in the LegalBench repository?
    • Every task is public domain because it appears in a repository
    • The repository’s top-level license applies to all third-party datasets without exception
    • Tasks draw on different source datasets, and their use terms may differ
    • A task’s instructions establish the current law in every jurisdiction
  8. A model scores well on public legal reasoning tasks. Which next step tests whether it fits a real workflow?
    • Repeat evaluation on another public split and treat it as representative of the target workflow
    • Tune prompts on the held-out examples, then report the same set as independent evaluation
    • Test only tasks that previously failed, then report the result as overall legal competence
    • Test on held-out examples that resemble the intended workflow and have legal experts inspect errors

Keep testing yourself

More quizzes picked for your level.

View all quizzes