What happened
Researchers tested whether coding agents remain reliable when repository code is changed in ways intended to preserve its meaning. Across two agent scaffolds, four models and two SWE-bench datasets, they report mostly small performance declines, but also substantial variation between system configurations.
An arXiv preprint submitted on Aug. 18, 2026, examines a practical reliability question: whether an AI code agent that can repair a repository-level issue on one codebase will remain effective after the surrounding code is rewritten into a semantically equivalent form. In this context, the rewritten code is intended to preserve program behavior while changing how that behavior is expressed. The paper describes the problem as poorly understood even as AI code agents are increasingly used to resolve software issues. The researchers introduce a random variant sampler that applies semantics-preserving transformations, or SPTs. The abstract names three classes of changes: control-flow rewrites, dead-code injection and identifier renaming. These alterations are designed to change the code’s surface structure without changing its intended semantics. The study then compares agent performance on unperturbed repositories with performance on the corresponding perturbed variants.
The evaluation covers two agentic scaffolds, mini-SWE agent and OpenCode. Each scaffold is paired with one of four models identified by the source as frontier models: Claude Opus 4.5, Kimi K2.5, MiniMax M2.5 and Qwen 3.6-27B. The test instances are drawn from SWE-bench Verified and SWE-bench Pro. For each instance, the agents are run multiple times on both the original and transformed versions, producing paired resolve-rate estimates. The pairing is intended to isolate the effect of the code transformation from the variability that comes from stochastic agent runs.
The paper reports small degradation in most configurations, with the largest average resolve-rate drop reaching 6.7 percentage points. The abstract says statistically significant degradation appears in six of the 16 combinations of model, scaffold and dataset. It also reports that no single model ranking by robustness holds across the two scaffolds. Qwen is described as among the most robust with mini-SWE agent on SWE-bench Verified, but as the most brittle with OpenCode. The simpler mini-SWE agent scaffold is reported to be more robust to the perturbations. The supplied source does not provide the exact number of benchmark instances, confidence intervals, per-transformation results or individual failure examples.
Read the primary source: arxiv.org ↗
Why it matters
The findings suggest that a coding agent’s success may depend partly on superficial properties of a repository, not only on the underlying software problem. That matters for organizations using agents to modify real code, although the supplied source does not establish production failures or independent replication.
The central implication is that code-agent reliability may not be determined solely by the underlying issue an agent is asked to fix. If two repositories preserve the same semantics but produce different success rates, then an agent can be sensitive to representation: naming, control-flow structure or the presence of irrelevant code. That would make performance less predictable when teams inherit unfamiliar repositories, refactor existing systems or use automated tools across codebases with different conventions.
The reported maximum decline of 6.7 percentage points is not evidence that every coding agent is broadly unreliable. The abstract says most configurations experienced small degradation, and statistically significant effects were found in six of 16 configurations. The result is therefore uneven rather than universal. Its importance lies in showing that a seemingly harmless change can measurably alter benchmark performance in at least some tested systems, while the size and direction of the effect depend on the model, scaffold and dataset.
The scaffold result is especially relevant to how coding agents are built. The paper reports that mini-SWE agent, characterized in the abstract as the simpler scaffold, was more robust to perturbation than OpenCode. That suggests the surrounding orchestration—such as how an agent reads a repository, plans work and uses tools—may influence robustness as much as the underlying model. The study’s cross-scaffold reversal in model rankings also cautions against treating a model’s benchmark position as a fixed property independent of the system in which it operates.
For organizations, the practical lesson is provisional: evaluations based only on a single canonical version of a repository may miss sensitivity to equivalent code forms. Testing transformed variants, retaining automated tests and requiring review of agent-generated changes could help expose that sensitivity. Those are operational implications, not outcomes demonstrated by this preprint. The supplied source reports benchmark experiments, not confirmed incidents, security breaches, financial losses or measured effects on production software. It also does not establish that any tested model is representative of all deployed coding agents.
What to watch next
The key follow-up is whether the reported pattern survives broader testing and detailed scrutiny of the benchmark instances, transformations and statistical results. Developers should watch for evaluations that test equivalent code variants, compare agent scaffolds and measure whether benchmark differences translate into operational errors.
The next point of scrutiny is the paper’s detailed experimental record. The abstract does not state how many SWE-bench Verified and SWE-bench Pro instances were used, how the transformations were sampled or distributed, or whether some types of rewrite were more disruptive than others. Readers should look for per-model and per-scaffold resolve rates, uncertainty intervals, statistical-test details and the practical size of each effect. Those details will determine whether the headline 6.7-point decline reflects a broad pattern or a small number of sensitive cases.
Replication will matter because the source identifies this as an arXiv submission and provides no independent validation. Follow-up studies could test the same method on additional coding agents, model versions, repository types and programming languages, while preserving the paired comparison between original and semantically equivalent code. They should also examine whether the transformations remain semantics-preserving for the specific build systems, tests and dependencies used by each repository.
A second question is whether benchmark sensitivity predicts real-world consequences. The paper does not report production telemetry or show that transformed code causes more incorrect patches, regressions, security vulnerabilities or review work outside the benchmark. Future evaluations should connect resolve-rate changes to patch correctness, test coverage, rollback rates and human review outcomes. They should also examine whether agents fail because they miss relevant code, misunderstand the task, misuse tools or produce patches that pass limited tests but do not solve the underlying issue.
Finally, evaluation designers may increasingly treat representation robustness as a separate dimension of agent quality. A useful assessment would compare several equivalent repository forms, multiple scaffolds and repeated runs, rather than relying on one model score or one code presentation. The current source supports that as a direction for testing, but does not identify a definitive robustness standard or show which mitigation works. Until those questions are answered, the paper’s finding is best read as evidence of an uneven reliability boundary, not as a complete measure of coding-agent capability.
