What happened
Scale AI and the Infocomm Media Development Authority (IMDA) announced the SG‑LegalClarify , a test suite designed to assess AI models’ legal reasoning under Singapore law and their ability to recognise when additional factual information is required. The benchmark comprises 1,100 English‑language legal questions across ten domains, with 760 of them deliberately omitting essential facts. For each omitted‑fact question, a counterpart version supplies the missing information, creating 1,860 evaluation instances. Models are prompted to ask for missing facts, and a simulated user provides the information when requested. The evaluation grades models on issue identification, rule recall, rule application and conclusion quality. Scale evaluated 19 models and reported that average mandatory‑criterion scores rose from 53.8% to 63.0% when web search was enabled, though clarification remained a challenge. The technical report details 12,546 lawyer‑written rubric items and a three‑model judging panel calibrated against human assessments. Scale plans to release a public subset of the benchmark later.
Scale AI and Singapore’s Infocomm Media Development Authority (IMDA) introduced SG‑LegalClarify on 12 November 2026. The is intended to test two core capabilities of AI models: (1) legal reasoning under Singapore law and (2) the ability to recognise when a question lacks sufficient factual context and to request that information.
The test set contains 1,100 English‑language questions written by practising lawyers, covering ten distinct legal domains such as contract law, torts, and intellectual property. Of these, 760 questions deliberately omit key facts needed for a definitive answer. For each omitted‑fact question, a paired version supplies the missing facts, yielding a total of 1,860 evaluation instances that allow direct comparison of model performance with and without fact‑elicitation.
During evaluation, a simulated user interacts with the model: when the model asks for missing information, the user provides the omitted facts. Model answers are graded on four criteria—issue identification, rule recall, rule application and conclusion—using a rubric of 12,546 items authored by lawyers. A three‑model judging panel, calibrated against human assessments, performed the grading.
Scale evaluated 19 AI models on the . The report notes that enabling web search raised the average mandatory‑criterion score from 53.8 % (no search) to 63.0 %, indicating that retrieval‑augmented capabilities can improve performance, though models still struggled with clarification tasks.
The collaboration between Scale and IMDA was formalised in April 2026, with an earlier public commitment announced in July 2026. The November release includes a technical report describing the design and initial findings, and Scale says a public subset of the benchmark will be released in the future.
Why it matters
The SG‑LegalClarify marks a concrete step toward systematic, domain‑specific evaluation of AI legal reasoning, an area that has long lacked standardized testing. By focusing on Singapore law—a civil‑law system distinct from common‑law jurisdictions—the benchmark expands the geographic and legal diversity of AI assessment, encouraging developers to build models that can handle varied statutory frameworks. The inclusion of fact‑elicitation tests pushes models beyond static question‑answering toward interactive, information‑gathering behavior, a capability critical for real‑world legal assistance where users often provide incomplete details. The reported improvement when web search is enabled highlights the importance of retrieval‑augmented generation for legal tasks, suggesting that future systems may need tighter integration with up‑to‑date legal databases. Public release of a benchmark subset will give researchers and firms a common yardstick, potentially accelerating progress, fostering transparency, and informing regulators about AI readiness for legal support roles.
Standardised evaluation is essential for measuring progress and ensuring safety in AI‑driven legal assistance. SG‑LegalClarify provides the first systematic, domain‑specific for Singapore law, filling a gap where most existing legal‑AI tests focus on U.S. or European contexts.
The fact‑elicitation component pushes models toward interactive reasoning, mirroring real‑world legal consultations where clients often provide incomplete information. Success in this area could broaden the practical applicability of AI tools in legal clinics, corporate compliance, and public‑sector advisory services.
The observed performance boost from web search underscores the growing relevance of retrieval‑augmented generation (RAG) for legal tasks, where up‑to‑date statutes, case law, and regulations are critical. This may drive more developers to integrate specialised legal databases and real‑time search into their models.
By planning a public subset, Scale and IMDA enable independent researchers, startups, and established vendors to benchmark their systems on a common footing, fostering transparency and competition.
Regulators can use results to gauge the readiness of AI systems for deployment in legal contexts, informing policy on licensing, liability, and consumer protection.
Interactive Mechanism: How It Actually Works
Explore the underlying technology behind this development interactively.
Which component of an AI application is the machine-learning model itself?
What to watch next
Watch for the public release of the SG‑LegalClarify subset, which will enable broader community testing and comparison. Monitor how AI developers incorporate retrieval‑augmented techniques to improve fact‑elicitation and rule‑application scores. Track any follow‑up studies from Scale or IMDA that examine model performance across additional legal jurisdictions or that expand the benchmark to cover multilingual questions. Finally, observe regulatory responses in Singapore and elsewhere as standardized legal‑AI benchmarks inform policy on AI‑assisted legal services.
The timing and scope of the public release – which questions and domains will be included, and whether multilingual or non‑English versions will follow.
Adoption of retrieval‑augmented architectures by model developers aiming to improve fact‑elicitation scores, potentially leading to new product features or API offerings.
Follow‑up studies that expand the to other jurisdictions (e.g., Malaysia, Hong Kong) or that test multilingual capabilities, which would broaden the benchmark’s relevance.
Regulatory commentary from Singapore’s legal authorities or international bodies on the implications of AI‑assisted legal reasoning, especially concerning professional standards and consumer protection.
Community‑driven leaderboards or competitions built around SG‑LegalClarify, which could accelerate innovation and surface best‑practice approaches.