LegalBench and Evaluating AI on Legal Tasks

LegalBench is an open-science benchmark assembled collaboratively by lawyers, legal researchers, and computer scientists. The authors describe 162 tasks covering six types of legal reasoning and report empirical evaluation of 20 open-source and commercial language models. The repository includes task datasets and instructions, with task-specific licenses that users must follow. Tasks include different input and output forms, such as legal questions, evidence descriptions, and passages to interpret.

Benchmark performance depends on task design, prompts, scoring, data split, model version, and whether examples have appeared in training. LegalBench’s breadth is useful for comparing task-specific behavior, but its authors present it as a way to study what types of reasoning models perform, not as a certification of legal practice. A result on hearsay classification does not show competence in drafting, jurisdiction-specific advice, negotiation, or fact investigation. Even a high score may conceal errors on particular subgroups or input formats.

For a meaningful evaluation, state the task subset, prompt, model version, scoring method, and data provenance. Check licenses and instructions, compare baselines, inspect errors, and use held-out examples from the intended workflow. Human legal experts should evaluate outputs when the intended use has legal consequences. Do not present one benchmark score as proof that a system is safe, accurate, or ready to replace professional judgment. The repository provides task prompts and answer guides alongside datasets, allowing evaluators to inspect how each task is scored and what source material it uses.