Guide-linked quiz · Medium Level
LegalBench and Evaluating AI on Legal Tasks Quiz
Tests LegalBench scope, task variety, metric limits, contamination, licensing, and how to interpret model results without overclaiming legal competence.
Question 1 of 8
What does the LegalBench paper describe?
Preview every question8 questions
Answers and explanations appear as you take the quiz.
- What does the LegalBench paper describe?
- A certification that a model can replace a law firm
- A collaborative benchmark of 162 tasks spanning six types of legal reasoning
- A benchmark covering only clause extraction in commercial contracts
- A database that searches all federal and state statutes for citations
- A model performs well on a hearsay-classification task. Which conclusion is justified?
- It can apply current hearsay law in every jurisdiction and factual setting
- It performed well on that task under the evaluation setup
- It matches lawyer performance across all benchmark categories
- It can advise clients on evidence without reviewing the underlying facts
- Why should an evaluator report results by task instead of only one aggregate score?
- A single overall score remains representative even when task types are imbalanced
- One overall score necessarily predicts success in a legal workflow
- Task-specific results are unnecessary when a benchmark has many examples
- Different tasks have different formats and scoring, so an average may hide uneven behavior
- What can happen when a benchmark is public and widely used?
- Public task materials may overlap with examples a model encountered during training
- Contamination occurs only when task answers are used in supervised fine-tuning
- Expert-authored tasks cannot have appeared in pretraining data
- Public benchmark exposure does not affect performance if prompts are hidden
- Which details make a LegalBench result reproducible?
- Task subset, prompt, model version, scoring method, and data provenance
- The data source and task list, but no prompt or scoring instructions
- The model version and runtime, but not task subset or answer normalization
- The legal topic and the highest score from a previous evaluation
- Before comparing two systems on LegalBench, what should the evaluator check?
- Same task data with different answer normalization across systems
- Same task, prompt, scoring, and evaluation data for both systems
- Same model API but different prompts and decoding settings
- Same benchmark name while allowing each team to select its own test subset
- Why inspect task-specific instructions and licenses in the LegalBench repository?
- Every task is public domain because it appears in a repository
- The repository’s top-level license applies to all third-party datasets without exception
- Tasks draw on different source datasets, and their use terms may differ
- A task’s instructions establish the current law in every jurisdiction
- A model scores well on public legal reasoning tasks. Which next step tests whether it fits a real workflow?
- Repeat evaluation on another public split and treat it as representative of the target workflow
- Tune prompts on the held-out examples, then report the same set as independent evaluation
- Test only tasks that previously failed, then report the result as overall legal competence
- Test on held-out examples that resemble the intended workflow and have legal experts inspect errors
Keep testing yourself
More quizzes picked for your level.