What happened
Together AI published a head-to-head evaluation of GLM-5.3 and GPT-5.6 Sol on the DeepSWE software-engineering benchmark. Sol solved more tasks on the first attempt; GLM-5.3 pulled ahead once multiple attempts were allowed, at about half the cost per rollout. A cascade that runs GLM-5.3 first and escalates failures to Sol scored highest of all.
Together AI, a company that sells inference and GPU capacity for open models, published a blog post on 21 August 2026 comparing GLM-5.3 against GPT-5.6 Sol on DeepSWE, a benchmark of software-engineering tasks spanning multiple languages and task types. The authors, listed as Zain Hasan and Shobhit Dixit, say they ran all 113 tasks four times per model with both systems at maximum effort, for 904 rollouts in total and 452 per side, drawing on the published per-trial records of a DeepSWE v1.1 export rather than running the harness end to end themselves.
On the headline metric, Sol led: 72.7% pass@1 against GLM-5.3's 69.0%, a 3.7-point gap the post itself says 'sits inside a couple of standard deviations' given reported error bars of ±2.2 and ±2.7 points. The order reverses with retries. At two attempts the two are level (81.1% for GLM-5.3, 81.0% for Sol); at four, GLM-5.3 leads 87.6% to 85.8%. Scoring errors as failures barely moves either figure (68.8% and 72.3%), because the run logged only one infrastructure error for GLM-5.3 and two for Sol.
The cost and speed picture splits the other way. Together AI reports $3.99 per GLM-5.3 rollout against $8.37 for Sol — 2.1 times cheaper, or 17 solved tasks per $100 against nine — with run totals of $1,806 and $3,783. Sol was the faster and terser worker: an average 19 minutes and 61 steps per rollout, and 60,000 output tokens, against GLM-5.3's 35 minutes, 124 steps and 80,000 output tokens. Sol was also more consistent, solving 61 tasks on all four trials to GLM-5.3's 48, and scoring 84.5% reliability against 78.8%.
The post's failure analysis is where the two diverge most usefully. Sol broke a repository's existing test suite in 20% of its failures; GLM-5.3 did so in 11%, and more often produced a near miss with the baseline intact (61% against 54%). By domain the board splits four wins each, with Sol stronger on exact-contract and systems work and GLM-5.3 stronger on query and configuration languages, runtime internals and stateful reactivity; GLM-5.3's clear weak spot is protocol conformance at 44%, fifteen points behind Sol. By language, GLM-5.3 leads on JavaScript (90 to 75) and Rust (70 to 60), and Sol on Python, Go and TypeScript.
Because per-task agreement between the two is only 0.43, their union covers 106 of the 113 tasks. Together AI's central recommendation follows from that: run GLM-5.3 first and escalate to Sol only when a test suite rejects the answer. That cascade, it reports, solves 85.9% of tasks at $6.61 each — above Sol alone and below the price of a single Sol rollout — and beats a hypothetical perfect one-shot router at 83.8%. The method note flags real limits: per-turn trajectory files for the GLM-5.3 batch were not on the public CDN, so the analysis is index-level; task domains were classified by a language model; and the cascade figure assumes a reliable verifier and independence between attempts.
Read the primary source: together.ai ↗
Why it matters
The result puts a concrete price on the remaining gap between an open-weight coding model and a closed frontier model, and shows that the ranking depends heavily on which metric a team optimizes. It also quantifies a practical safety difference: how often each model breaks tests that already passed.
For teams choosing models for agentic coding work, this is one of the more specific public accountings of what a frontier model's premium buys. The claim is not that the open model is better, but that the gap on first-attempt accuracy is under four points and inside the reported noise, while the price gap is a factor of two. Expressed as work per dollar — 17 solved tasks per $100 against nine — the comparison reframes model selection as a budget question rather than a capability question for a large share of routine tasks.
The result also shows how much the answer depends on the metric. A team that must accept the first output, with a human waiting, should read Sol's higher pass@1, higher reliability and roughly half the wall-clock time as decisive. A team running batch jobs behind a test suite should read GLM-5.3's pass@4 and coverage lead the same way. These are not competing interpretations of one number; they are different numbers, and the post is explicit that best-of-k advantages only convert into value if something can automatically tell a good patch from a bad one.
The failure anatomy matters more than the headline for anyone shipping model-written diffs. A model that breaks previously passing tests in 20% of its failures imposes a different review burden than one that does so in 11%, regardless of which scores higher. Together AI's practical advice — put a full regression run around Sol's output before accepting it — is the kind of operational detail that benchmark leaderboards usually omit, and it is checkable by any team that runs its own suite.
Several caveats limit how far the findings travel. This is a single benchmark of 113 tasks measured by a vendor whose business benefits when open models look competitive; it has not been independently reproduced, and the post says its own figures may differ from other public scorecards of the same pair. Costs depend on prices that can change without notice. The post calls GLM-5.3 'the open-weight challenger' but does not state its developer, license, or where the weights can be obtained; AI Understanding has previously reported Zhipu AI open-sourcing GLM-5.2, and it is not established here that GLM-5.3 is available on the same terms. Neither OpenAI nor the GLM developer is quoted, and no response from either appears in the source.
What to watch next
Whether independent groups reproduce the numbers, whether the reported price gap survives pricing changes, whether GLM-5.3 weights are in fact broadly available, and whether verifier-gated cascades become a standard deployment pattern rather than a benchmark artifact.
The first thing to watch is replication. Together AI worked from published per-trial records and says the per-turn trajectory files for its GLM-5.3 batch were missing from the public CDN when it ran the analysis, which kept the work at index level. If those artifacts are posted, other researchers can check the failure-anatomy and near-miss classifications, which rest on definitions the post supplies (a near miss is at least 80% of new tests passing with the baseline intact) rather than on benchmark-standard measures.
The second is whether the cascade result survives contact with real repositories. Its 85.9% figure assumes a verifier good enough to decide when to escalate, and that the second attempt is independent of the first. Projects with thin or slow test suites do not have that verifier, and a test suite that passes a wrong patch turns the cascade into a cheap-model-only pipeline. Verifier-gated routing has become a visible product category this year, and this analysis gives one testable claim about how much it can actually recover.
The third is durability of the price gap. Both the 2.1x cost ratio and the per-$100 solve counts are functions of current inference pricing and of running both models at maximum effort; lower effort settings, quantized serving, or a price change on either side could reorder the table without any change in model quality. The same applies to the speed gap, which reflects how many steps each model takes rather than any fixed property of the underlying systems.
Finally, watch the specific weaknesses each side shows. GLM-5.3's 44% on protocol conformance is the one domain where the post reports a decisive Sol advantage, and it is the sort of gap a point release can close or entrench. Whether subsequent versions of either model shift those per-domain and per-language splits — and whether DeepSWE itself stays uncontaminated as it is used more widely — will determine how long routing rules derived from this run remain useful.
