What happened
A single-author preprint posted to arXiv on August 12, 2026 introduces SemPlan Benchmark, a deterministic synthetic test set of 1,800 English and Brazilian Portuguese cases, and compares four architectures for turning natural-language requests into governed database queries. Answer correctness ranged from 22.25% to 25.67%, with different architectures leading on safety, cost and refusal behavior.
A preprint posted to arXiv on August 12, 2026 under the identifier 2608.13612, authored by Bruno Santos Teixeira, describes a benchmark called SemPlan aimed at a narrow but commercially common problem: how a system should turn an underspecified natural-language request into a query that runs against enterprise data without producing invalid results, violating access policy, running up cost, or returning different answers each time it is asked. The abstract frames the work as an evaluation of an architectural design space rather than of any particular model.
The benchmark is described as deterministic, synthetic and bilingual, containing 1,800 cases in English and Brazilian Portuguese, of which 1,200 form what the author calls a frozen scientific evaluation subset. Four architectures are compared under what the abstract describes as the same model configuration: direct SQL generation (labeled A1); a bounded tool-agent baseline (A2); structured semantic-request generation followed by deterministic planning and execution (A3); and a clarification-oriented, stateful semantic-plan variant (A4).
Across what the abstract calls 4,800 primary records, answer correctness was low in absolute terms for every design: 22.25% for A1, 22.58% for A2, 25.67% for A3 and 24.25% for A4. The record count is consistent with running the 1,200-case frozen subset through all four architectures, though the abstract does not state that mapping explicitly. A3 had the highest observed correctness and, according to the abstract, significantly exceeded A1, A2 and A4 in a pre-specified paired correctness analysis — a design choice that means the comparison was planned before the results were seen rather than selected afterward.
The other reported metrics do not line up behind the same winner. A1, the simplest approach of generating SQL directly, retained the highest policy-correct rate and the lowest unsafe-or-invalid rate. A4, the clarification and state-tracking variant, had the lowest mean API cost and the lowest false-refusal rate — that is, it least often declined a request it should have answered. On a preselected 150-case stability subset, answer-correct repeatability ranged from 92.00% to 98.67%, indicating that none of the architectures produced fully deterministic answers even when the same case was repeated.
Several things the abstract does not establish are worth stating plainly. It does not name the language model used, describe the database schemas or the policy rules being enforced, define how policy-correct, unsafe-or-invalid and false-refusal were scored, give the actual cost figures behind the cost comparison, or break correctness down by language despite the bilingual design. The listing page does not indicate that code or data have been released. The paper is an 11-page preprint with three figures and nine tables from one author; the abstract says it has been submitted to Transactions on Machine Learning Research, which means it has not completed peer review.
Read the primary source: arxiv.org ↗
Why it matters
Natural-language interfaces to company data are one of the most widely sold enterprise AI features, and the paper's central claim is that adding structure and planning layers changes which failures happen rather than eliminating them. The absolute scores come from a synthetic benchmark of the author's own construction, so they are not a measurement of production accuracy.
Chat-style access to internal databases is among the most heavily marketed enterprise AI capabilities, and vendors typically differentiate on architecture: whether the model writes SQL directly, calls a constrained set of tools, emits a structured intermediate representation that a deterministic planner executes, or asks the user clarifying questions. This paper takes that marketing distinction and tests it under fixed conditions. Its headline finding is that the architecture choice reshapes the failure profile without reliably fixing accuracy.
The absolute correctness numbers deserve care. They come from a synthetic benchmark whose difficulty was set by its own author, so a score in the low twenties is a property of this test set, not an estimate of how often a deployed system answers a real business question correctly. What the design does support more strongly is the relative comparison: because all four architectures ran under the same model configuration on the same frozen cases, the ordering between them is more informative than the raw percentages.
The trade-off pattern is the practically useful part for anyone evaluating these systems. The design that answered most questions correctly was not the one that best respected policy, and the one that was cheapest and least likely to refuse valid work was neither the most correct nor the safest. A buyer who selects on a single accuracy figure would, on this evidence, be choosing without seeing the governance and cost consequences. The paper's framing — a trade-off interpretation rather than a universal ranking — is a caution against claims that any one architecture solves the problem.
The repeatability result speaks to a separate operational concern. Answer-correct repeatability between 92.00% and 98.67% on a 150-case subset means that repeating the same question sometimes changed whether the answer was right. For reporting, auditing or compliance workflows, where the same query is expected to produce the same figure, that variability is a distinct risk from average accuracy and is rarely reported in vendor materials.
The bilingual construction, covering English and Brazilian Portuguese, addresses a gap in benchmarks that test English only. Whether the two languages performed comparably is not stated in the abstract, so the design's value here is currently potential rather than demonstrated. More broadly, the study's weight is limited by being a single-author preprint on synthetic data with one undisclosed model configuration; it is a structured hypothesis about architecture trade-offs, not a settled result.
What to watch next
Whether the full paper names the model used, breaks results down by language, and defines its safety metrics; whether the benchmark code and data are released so other groups can rerun it on different models; and whether the paper clears peer review at TMLR, where the abstract says it has been submitted.
The most immediate question is what the full paper contains. Which model was used, how the policy and safety metrics were defined and scored, what the cost figures actually were, and whether results differed between English and Brazilian Portuguese are all unresolved from the abstract alone. The nine tables the paper reports may answer several of these; the abstract is not sufficient to judge whether the safety and refusal claims are measured consistently across architectures.
Whether the benchmark itself is released matters more than the specific percentages. The frozen 1,200-case subset is designed for reuse, and its value depends on other groups being able to run it against models the author did not test. If the code and data are published and the trade-off pattern reproduces across several frontier models, the finding becomes considerably harder to dismiss. If it does not reproduce, or if the results turn out to hinge on one model's behavior, the architectural claim narrows sharply.
Peer review is a concrete near-term checkpoint. The abstract states the paper has been submitted to Transactions on Machine Learning Research; acceptance, rejection or a revised version would all be informative, particularly on whether the pre-specified paired analysis holds up under reviewer scrutiny given how close the correctness figures are to one another.
Longer term, the useful development would be benchmarks of this shape built on real enterprise schemas and real access-control policies rather than synthetic ones, and vendors reporting policy-correctness, false-refusal rate and repeatability alongside accuracy. Until that happens, buyers evaluating natural-language data interfaces have limited public evidence to compare products against, and this paper is best read as a template for what to ask a vendor rather than as a verdict on any product on the market.
