What happened
A new study by researchers at Shandong University compared three production AI coding harnesses—Claude Code, OpenCode, and a minimalist 100-line Python script called mini-SWE-agent—using five different underlying models. The research demonstrated that while harness choice significantly impacts inference costs, it does not consistently improve accuracy. On a set of 447 software engineering tasks, the most elaborate harness (Claude Code) and the minimalist harness (mini-SWE-agent) produced statistically equivalent success rates. However, cost per task varied by as much as threefold, with Claude Code being the most expensive due to larger initial context sizes and repeated transmission of reasoning blocks.
Researchers at Shandong University conducted a controlled experiment to isolate the impact of the 'harness'—the , tools, and context management rules—on AI coding performance. They held the underlying models constant and tested three distinct harnesses: Claude Code (elaborate), OpenCode (intermediate), and mini-SWE-agent (minimalist, ~100 lines of Python). The models included two local open-weight Qwen models and three API-served models (DeepSeek, GLM, and HY4), with Claude Opus 5 used as a top-tier anchor in Claude Code.
The evaluation took place in isolated containers across 492 offline tasks from SWE-bench Verified, including 45 particularly difficult tasks. The results showed that on the larger 447-task pool, Claude Code and mini-SWE-agent achieved statistically equivalent success rates within a five-percentage-point margin. On the 45 hardest tasks, switching harnesses changed which problems were solved about as often as simply rerunning the same setup, with a median switch rate of 13%.
While accuracy remained largely unchanged, cost per task varied dramatically. On the GLM-5.3-Flash model, a trial in Claude Code cost nearly twice as much as in mini-SWE-agent and over three times as much as in OpenCode. This cost disparity stemmed from the initial context size: Claude Code sent 16,581 tokens in its first call, compared to 7,025 for OpenCode and 829 for mini-SWE-agent. Because agents resend the conversation transcript on every turn, this initial overhead is multiplied across the entire task.
The study also highlighted the unreliability of single-run benchmarks. The researchers noted that 10 reruns of the exact same setup on SWE-bench Verified can yield score spreads of 2.2 to 6.0 percentage points. They found that 71% of early web-enabled trials had retrieved the original human-written fix from GitHub, which they discarded to ensure a fair offline evaluation. The only harness effect that cleared the statistical noise floor was a loss: OpenCode trailed by up to 9 points on the large task pool, largely due to hitting output token limits without automatic continuation.
Source details: venturebeat.com ↗
Why it matters
This research challenges the common industry assumption that more complex agent scaffolding leads to better performance. For enterprise leaders and developers, the findings indicate that vendor benchmarks based on single runs are unreliable due to high run-to-run variance. The study provides a practical framework for evaluating coding agents by focusing on cost-per-task and statistical significance rather than isolated accuracy spikes. It highlights that the 'harness tax'—the extra cost incurred by complex system prompts and tool schemas—can be substantial without delivering proportional value, urging teams to their specific model-harness combinations before adopting expensive, feature-rich agents.
The findings directly contradict the marketing narratives of many AI vendors who advertise accuracy gains from harness updates. By demonstrating that a 100-line script can match the performance of a complex commercial agent, the study suggests that the 'harness tax' is often a cost inefficiency rather than a performance necessity. This is critical for enterprises allocating budgets for AI coding tools, as it implies that significant savings can be achieved by stripping down turn-one preambles and tool schemas without sacrificing task completion rates.
The research provides a methodological correction for the industry: single-run comparisons are insufficient to determine harness superiority. With a 13% task flip rate on hard problems, a 45-task evaluation suite can only catch a 13-point gap half the time. This means that small, single-run differences often reflect random variation rather than true capability differences. Teams are advised to run baseline configurations multiple times to establish a reliable noise floor before adopting more complex scaffolds.
The study also underscores the importance of model-harness interaction. For instance, turning off extended thinking for Qwen3.6 caused an 11-point drop in pass rate for mini-SWE-agent, but changes within run-to-run variation for Claude Code. This indicates that cost-optimization strategies, such as adjusting reasoning budgets, must be tailored to the specific model-harness pair rather than applied generically. The practical implication is that engineering leaders need to their exact stack to make informed decisions about tooling and budget allocation.
Interactive Mechanism: How It Actually Works
Explore the underlying technology behind this development interactively.
An agent must create a draft calendar event for Tuesday at 2 p.m. Which evidence would establish the requested result?
What to watch next
Developers and enterprises should monitor how AI vendors respond to these findings, potentially by offering more transparent cost breakdowns or leaner harness options. There may be a shift in the market toward minimalist agent architectures that prioritize cost efficiency over feature bloat. Additionally, watch for updated benchmarking methodologies that account for run-to-run variance and prompt caching effects, which could change how coding agent performance is evaluated in the industry.
Expect AI vendors to refine their benchmarking methodologies to address the variance issues highlighted in the study. This may include publishing confidence intervals or requiring multiple runs for leaderboard entries. Vendors may also introduce 'lean' modes or configurable harnesses that allow users to trade off features for cost efficiency, directly responding to the 'harness tax' findings.
The open-source community may see a rise in minimalist harnesses that challenge the dominance of feature-rich commercial agents. If developers can achieve comparable results with simpler, cheaper tools, it could shift the competitive landscape away from proprietary, complex scaffolds toward transparent, efficient alternatives. This could also influence how AI models are trained and optimized, with a greater focus on reducing the need for elaborate context management.
Enterprise adoption of AI coding agents may become more cautious, with a stronger emphasis on total cost of ownership (TCO) rather than just accuracy. Organizations may develop internal standards for evaluating agent performance that include cost-per-task metrics and statistical significance tests. This shift could lead to a more mature and rational market for AI development tools, where value is measured by practical outcomes and efficiency rather than marketing claims.