What happened
Researchers introduced an open-source benchmark for diagrammatic reasoning in olympiad geometry. It contains 954 self-contained problems, including a 297-problem hard subset, paired with solutions and human-authored diagrams represented in renderable Asymptote code. The paper reports that current foundation models often solve geometry problems more reliably than they construct accurate diagrams.
An arXiv paper titled “Solving Is Not Drawing: A Benchmark for Diagrammatic Reasoning in Olympiad Geometry” presents a new evaluation of foundation models. The record lists Hsien Xin Peng, Anthony Kim, Alvin Li, Calvin Supasanya, Shivank Garg and Kevin Zhu as authors, and says the paper was submitted on June 23, 2026. It also lists ICML 2026 and the AI4MATH Workshop in the comments, but the supplied source does not establish whether the work was formally accepted. The paper’s central claim is that solving a geometry problem and constructing the figure it depends on are different abilities.
The benchmark includes 954 self-contained olympiad geometry problems and a 297-problem hard subset. Each problem is paired with its solution and a human-authored, high-fidelity diagram encoded in renderable Asymptote code. This design gives researchers a reference representation against which model-generated diagrams can be assessed. The source describes the collection as open source and says it can be accessed through a link, but the supplied arXiv record does not identify the repository, license terms or access conditions in more detail.
The authors describe a suite of text-, code-, image-, vision-language-model and constraint-based metrics for evaluating what they call diagrammatic reasoning. These metrics are intended to assess more than whether a model reaches a final mathematical answer: they target whether it can construct a diagram with appropriate geometric relationships and auxiliary elements. The abstract does not define each metric, explain how the different measurements are combined, or say whether human experts validated the scores. Those details are therefore important unresolved parts of the study.
In the paper’s reported evaluation, current foundation models produced diagrams that were “markedly less faithful” than their mathematical-solving ability would suggest. The authors report an average compile success rate of 36.14%, meaning that a substantial share of generated diagram code did not successfully compile under the benchmark’s rendering setup. They conclude that strong mathematical reasoning does not imply accurate diagram construction. The supplied source does not name the evaluated models, state the number of trials, give model-by-model scores or report uncertainty around the average.
Read the primary source: arxiv.org ↗
Why it matters
The work isolates a capability that common mathematics benchmarks may not measure: producing a geometrically faithful figure with the necessary points, lines, circles and auxiliary constructions. That distinction matters wherever a model’s output must be inspected, rendered or used as part of a larger mathematical workflow. The reported results are limited to the benchmark and the evidence summarized in the arXiv abstract.
Geometry problems often depend on a figure that encodes relationships among points, lines, circles and angles. The source emphasizes that progress can hinge on a faithful diagram containing the right auxiliary constructions and incidences. A system may therefore demonstrate substantial problem-solving ability while still failing to produce a visual object that another person or program can inspect. Separating these tasks makes it possible to ask a more precise question about what a model actually knows and can operationalize.
Most widely used mathematics evaluations focus on whether a system reaches the correct answer. The benchmark described here adds a different layer: whether the system can express the relevant structure in a renderable diagram. That is potentially useful for research on multimodal reasoning, mathematical education and software that turns natural-language problems into visual explanations. The paper does not establish that the benchmark predicts classroom usefulness, proof quality or performance in professional mathematical work, so those practical connections remain to be tested.
The reported 36.14% compile success rate is a concrete warning about reliability at the implementation boundary. A diagram that cannot compile cannot be rendered for inspection, regardless of whether the model’s underlying reasoning was sound. At the same time, compilation is only one part of fidelity. Code can compile while placing an object incorrectly, omitting a needed construction or violating the intended incidence relations. The paper says it uses several additional metrics, but the abstract does not provide their results, so the compile figure should not be treated as a complete measure of diagram quality.
The work’s significance is also bounded by its domain. The benchmark covers olympiad geometry, a specialized setting with conventions and structures that may not represent scientific plots, engineering schematics, educational figures or general visual reasoning. The source does not report transfer to other domains, comparisons with human performance, or the effect of diagram style and problem wording. It also does not show whether the evaluated systems were general-purpose models, vision-language models, code-focused systems or other categories. Those unknowns make the finding a focused research result rather than evidence about all visual tasks performed by AI systems.
What to watch next
The key follow-up questions are whether the benchmark’s metrics track expert judgments of geometric fidelity, how performance varies across models and problem difficulty, and whether the reported gap persists outside olympiad geometry. The source does not identify the evaluated models, provide per-model results, describe the metric definitions in detail or establish how training and prompting affect performance.
A first test will be whether independent researchers reproduce the reported gap using the released benchmark and obtain similar results across the full 954-problem collection and the 297-problem hard subset. Useful reporting would include the exact models and versions, prompting or decoding settings, number of attempts, compile environments and performance broken down by problem type. Without those details, the average result is difficult to compare across studies or over time.
Researchers should also examine whether the benchmark’s metrics agree with expert assessments of geometric correctness. In particular, future analyses could separate syntax or compilation failures from semantic errors in which a rendered figure looks plausible but does not preserve the intended relationships. The source says the benchmark includes code-, image-, vision-language-model and constraint-based measurements, but does not show how those metrics rank individual outputs or whether they reach consistent conclusions. Metric validation will determine how much confidence to place in headline scores.
Performance on the hard subset and on different construction requirements may reveal whether failures arise mainly from code generation, visual grounding, geometric planning or the coordination of those abilities. It will also be important to test problems with unfamiliar layouts, varied wording and different diagram conventions. The supplied source gives no evidence about generalization beyond the benchmark’s own problems, so results on newly authored tasks and non-olympiad geometry would be especially informative.
Finally, follow-up work can test whether targeted training, tool use, structured intermediate representations or iterative verification narrows the gap between solving and drawing. A model that generates a diagram and checks its own incidences may behave differently from one that writes a single block of Asymptote code. The practical question is not only whether models improve their compile rate, but whether they produce diagrams that people can trust and understand. The source does not report such interventions or user studies, leaving their effectiveness and costs unknown.
