Útmutatóhoz kapcsolódó kvíz · Közepes Szint

LegalBench and Evaluating AI on Legal Tasks Quiz

Tests LegalBench scope, task variety, metric limits, contamination, licensing, and how to interpret model results without overclaiming legal competence.

Kapcsolódó útmutató utakLegalbench Evaluating AI Legal Reasoning
kérdés 1 -ból 8

What does the LegalBench paper describe?

Tekintse meg az összes kérdést8 kérdéseket

A válaszok és magyarázatok megjelennek a kvíz kitöltésekor.

  1. What does the LegalBench paper describe?
    • A certification that a model can replace a law firm
    • A collaborative benchmark of 162 tasks spanning six types of legal reasoning
    • A benchmark covering only clause extraction in commercial contracts
    • A database that searches all federal and state statutes for citations
  2. A model performs well on a hearsay-classification task. Which conclusion is justified?
    • It can apply current hearsay law in every jurisdiction and factual setting
    • It performed well on that task under the evaluation setup
    • It matches lawyer performance across all benchmark categories
    • It can advise clients on evidence without reviewing the underlying facts
  3. Why should an evaluator report results by task instead of only one aggregate score?
    • A single overall score remains representative even when task types are imbalanced
    • One overall score necessarily predicts success in a legal workflow
    • Task-specific results are unnecessary when a benchmark has many examples
    • Different tasks have different formats and scoring, so an average may hide uneven behavior
  4. What can happen when a benchmark is public and widely used?
    • Public task materials may overlap with examples a model encountered during training
    • Contamination occurs only when task answers are used in supervised fine-tuning
    • Expert-authored tasks cannot have appeared in pretraining data
    • Public benchmark exposure does not affect performance if prompts are hidden
  5. Which details make a LegalBench result reproducible?
    • Task subset, prompt, model version, scoring method, and data provenance
    • The data source and task list, but no prompt or scoring instructions
    • The model version and runtime, but not task subset or answer normalization
    • The legal topic and the highest score from a previous evaluation
  6. Before comparing two systems on LegalBench, what should the evaluator check?
    • Same task data with different answer normalization across systems
    • Same task, prompt, scoring, and evaluation data for both systems
    • Same model API but different prompts and decoding settings
    • Same benchmark name while allowing each team to select its own test subset
  7. Why inspect task-specific instructions and licenses in the LegalBench repository?
    • Every task is public domain because it appears in a repository
    • The repository’s top-level license applies to all third-party datasets without exception
    • Tasks draw on different source datasets, and their use terms may differ
    • A task’s instructions establish the current law in every jurisdiction
  8. A model scores well on public legal reasoning tasks. Which next step tests whether it fits a real workflow?
    • Repeat evaluation on another public split and treat it as representative of the target workflow
    • Tune prompts on the held-out examples, then report the same set as independent evaluation
    • Test only tasks that previously failed, then report the result as overall legal competence
    • Test on held-out examples that resemble the intended workflow and have legal experts inspect errors