āĻĒāϰāĻŦāϰā§āϤ⧠āĻāĻĒāĻĒāϰāĻŦāϰā§āϤ⧠āĻāĻžāĻāĻĄ
LegalBench and Evaluating AI on Legal Tasks
āĻā§āĻāύāĻŋāĻā§āϝāĻžāϞ
āĻĒā§āϰāϝā§āĻā§āϤāĻŋāĻāϤ āĻāĻžāĻāĻĄ
CUAD is an expert-annotated dataset for contract-review NLP, not a complete representation of all agreements or legal questions.
āĻāĻ āύāĻŋāϰā§āĻĻā§āĻļāĻŋāĻāĻž āĻļā§āύā§āύāĻāĻāĻ-āĻŦāϰā§āĻŖāĻŋāϤ ¡ ~3 min
Its labels target 41 clause types in 510 commercial contracts; benchmark results measure performance on that task and distribution, not a systemâs ability to practice law.
The Contract Understanding Atticus Dataset (CUAD) was created for natural-language processing research in contract review. The Atticus Project describes version 1 as 510 commercial contracts with more than 13,000 expert-supervised labels covering 41 clause types considered important in corporate transactions, including mergers and acquisitions. The dataset was accepted at NeurIPS 2021 and is released under a stated license on the project page. CUAD frames clause review as finding or classifying spans of contract text. It can support research on clause detection, extraction, and related methods, but labels represent selected categories and annotation choices. A strong score does not establish that a system understands every legal interaction, that it performs well on other contract populations, or that its output is ready for a transaction. Contract styles, jurisdictions, languages, clause definitions, scanning quality, and amendments can all differ from the dataset. Before using CUAD, check its version, license, task definition, split design, and label handbook. Avoid leakage between training and test examples that share templates or related documents. Evaluate on separate agreements reflecting the intended use, inspect false positives and missed clauses, and have legal reviewers assess whether extracted spans preserve exceptions and context. CUAD is a research benchmark, not legal advice, a model certification, or an endorsement of automated contract review. Annotation instructions and adjudication also shape what counts as a correct span, so an evaluation should match the datasetâs exact task definition.
A researcher evaluates a clause-finding model against CUADâs annotated spans.
A reviewer checks whether a benchmark result transfers from commercial contracts to a new contract type.
A team studies which clause labels are represented before designing a contract-review model.
A data scientist separates development examples from held-out evaluation contracts.
Future contract benchmarks may add document types, languages, and richer measures of clause interaction. Such expansion can improve coverage but will not make a benchmark equivalent to legal practice. Teams should test against current agreements and keep experts involved in defining and validating outputs. Newer datasets may broaden contract types or model clause relationships, yet deployment still needs target-specific evaluation. Compare results against the actual agreements and business decisions in scope, and keep the model from silently converting benchmark labels into legal conclusions.
āĻĢā§āϰāĻŋ āύāĻŋāĻāĻāϞā§āĻāĻžāϰ
āĻĒā§āϰāϤāĻŋ āϏāĻĒā§āϤāĻžāĻšā§āϰ āĻāĻžāĻā§āϰ āĻĻāĻŋāύ āϏāĻāĻžāϞ⧠āϤāĻŋāύāĻāĻŋ āϝāĻžāĻāĻžāĻāĻā§āϤ āĻāĻāĻ āϏā§āĻā§āϰāĻŋ, āϏāĻšāĻ āĻāĻāϰā§āĻāĻŋāϤ⧠āϞā§āĻāĻžāĨ¤ āĻāĻŋāϰāĻāĻžāϞā§āϰ āĻāύā§āϝ āĻĢā§āϰāĻŋ, āĻā§āύ⧠āĻŦāĻŋāĻā§āĻāĻžāĻĒāύ āύā§āĻāĨ¤
āĻĒā§āϰāϤāĻŋ āϏāĻĒā§āϤāĻžāĻšā§āϰ āĻāĻžāĻā§āϰ āĻĻāĻŋāύ⧠āĻāĻāĻāĻŋ āĻāĻŽā§āϞāĨ¤ āĻāĻ āĻā§āϞāĻŋāĻā§āĻ āĻāύāϏāĻžāĻŦāϏā§āĻā§āϰāĻžāĻāĻŦ āĻāϰā§āύāĨ¤ āĻāĻŽāϰāĻž āĻāĻāύā§āĻ āĻāĻĒāύāĻžāϰ āĻ āĻŋāĻāĻžāύāĻž āĻŦāĻŋāĻā§āϰāĻŋ āĻŦāĻž āĻļā§ā§āĻžāϰ āĻāϰāĻŋ āύāĻžāĨ¤
āύāĻŋāĻā§āĻā§ āĻĒāϰā§āĻā§āώāĻž āĻāϰā§āύ
āĻĒā§āϰāϤāĻŋāĻāĻŋ āĻāϤā§āϤāϰā§āϰ āϤāĻžā§āĻā§āώāĻŖāĻŋāĻ āĻĢāĻŋāĻĄāĻŦā§āϝāĻžāĻ, āĻāĻŦāĻ āĻā§āϰā§āϏ āĻĒāĻžāϏ āĻāϰāĻžāϰ āĻĒāϰ āϝāĻžāĻāĻžāĻāϝā§āĻā§āϝ āĻāĻāĻĄāĻŋ āϏāĻš āĻāĻāĻāĻŋ āĻļā§ā§āĻžāϰāϝā§āĻā§āϝ āϏāĻžāϰā§āĻāĻŋāĻĢāĻŋāĻā§āĻāĨ¤
āĻĢā§āϰāĻŋ āĻāĻāĻ āĻļāĻŋāĻā§āώāĻžāĻā§ āϏāĻŽāϰā§āĻĨāύ āĻāϰā§āύāĨ¤ AI Understanding āĻāĻāĻāĻŋ 501(c)(3) āĻ āϞāĻžāĻāĻāύāĻ āĻĒā§āϰāϤāĻŋāώā§āĻ āĻžāύ â āĻā§āύ⧠āĻŦāĻŋāĻā§āĻāĻžāĻĒāύ āύā§āĻ, āĻā§āύ⧠āĻĒā§āĻā§āĻžāϞ āύā§āĻ, āĻāĻāύā§āĻ āύāĻžāĨ¤ āĻāĻāĻāĻŋ āĻĻāĻžāύ āĻāϰā§āύ
CUAD is an expert-annotated dataset for contract-review NLP, not a complete representation of all agreements or legal questions. Its labels target 41 clause types in 510 commercial contracts; benchmark results measure performance on that task and distribution, not a systemâs ability to practice law.
āĻĒā§āϰāĻāϞā§āĻĒāĻāĻŋ āϤāĻžāϰ āĻāĻāĻžāϰ, āϞā§āĻŦā§āϞ, āϧāĻžāϰāĻž āĻŦāĻŋāĻāĻžāĻ āĻāĻŦāĻ āĻŦāĻŋāĻļā§āώāĻā§āĻā§āϰ āϤāϤā§āϤā§āĻŦāĻžāĻŦāϧāĻžāύ āĻŦāϰā§āĻŖāύāĻž āĻāϰā§āĨ¤
āĻāĻžāĻāĻāĻāĻŋ āĻā§āĻā§āϤāĻŋ āĻĒāϰā§āϝāĻžāϞā§āĻāύāĻž āĻāύāĻāϞāĻĒāĻŋāϰ āĻāύā§āϝ āĻāĻāĻāĻŋ āĻŦāĻŋāĻļā§āώāĻā§āĻ-āĻā§āĻāĻžāϝā§āĻā§āϤ āĻĄā§āĻāĻžāϏā§āĻ āĻšāĻŋāϏāĻžāĻŦā§ CUAD āĻā§ āĻāĻĒāϏā§āĻĨāĻžāĻĒāύ āĻāϰā§āĨ¤
āĻāĻāĻāĻŋ āĻŦā§āĻā§āĻāĻŽāĻžāϰā§āĻ āϏā§āĻā§āϰ āĻĄā§āĻāĻžāϏā§āĻ, āĻāĻžāϏā§āĻ āĻāĻŦāĻ āĻŽā§āϞā§āϝāĻžāϝāĻŧāύ āĻĒā§āϰā§āĻā§āĻāϞ āĻĻā§āĻŦāĻžāϰāĻž āĻāĻŦāĻĻā§āϧāĨ¤
āĻāĻāĻžāϰāϞā§āϝāĻžāĻĒāĻŋāĻ āĻā§āĻŽāĻĒā§āϞā§āĻ āĻŦāĻž āϏāĻŽā§āĻĒāϰā§āĻāĻŋāϤ āύāĻĨāĻŋ āĻĒā§āϰāĻļāĻŋāĻā§āώāĻŖ āĻāĻŦāĻ āĻŽā§āϞā§āϝāĻžāϝāĻŧāύā§āϰ āĻŽāϧā§āϝ⧠āĻĢā§āĻā§ āϤā§āϰāĻŋ āĻāϰāϤ⧠āĻĒāĻžāϰā§āĨ¤
āĻŦāĻŋāĻāĻŋāύā§āύ āĻā§āĻā§āϤāĻŋāϰ āϧāϰāύ āĻāĻŦāĻ āĻāĻāϤāĻŋāϝāĻŧāĻžāϰ āĻāĻžāώāĻž āĻāĻŦāĻ āĻāĻžāĻā§āϰ āĻĒā§āϰāϝāĻŧā§āĻāύā§āϝāĻŧāϤāĻž āĻĒāϰāĻŋāĻŦāϰā§āϤāύ āĻāϰāϤ⧠āĻĒāĻžāϰā§āĨ¤
āĻļāĻŋāĻāϤ⧠āĻĨāĻžāĻā§āύ
āĻāĻ āĻŦāĻŋāώāϝāĻŧā§āϰ āĻāύā§āϝ āĻŦāĻžāĻāĻžāĻ āĻāϰāĻž āĻāϰāĻ āĻāĻžāĻāĻĄ
āĻĒāϰāĻŦāϰā§āϤ⧠āĻāĻĒāĻĒāϰāĻŦāϰā§āϤ⧠āĻāĻžāĻāĻĄ
LegalBench and Evaluating AI on Legal Tasks
āĻā§āĻāύāĻŋāĻā§āϝāĻžāϞ