Researchers from Shandong University analyzed the impact of coding agent harnesses, such as Claude Code, OpenCode, and mini-SWE-agent, on model performance. The study found that switching between these tools often yields results comparable to simply rerunning the same setup, with performance variations falling within the range of statistical noise.
The analysis involved five models, including Qwen3.6, Qwen3.8, Deepseek-V4-Flash, GLM-5.3-Flash, and HY4-Preview, tested on 45 difficult tasks and a pool of 447 tasks from SWE-bench Verified. While some developers claim significant performance leads for their harnesses, the researchers concluded that these differences are often statistically insignificant and comparable to random fluctuations.
The study highlights that while performance gains are minimal, costs vary significantly. Harnesses use different system prompts and tool descriptions, which are resent with every step. Claude Code, for instance, uses a system prompt of approximately 16,600 tokens, compared to under 1,000 for mini-SWE-agent. Consequently, a single task can cost up to three times more depending on the chosen harness.
OpenCode was the only harness to show a consistent performance deficit, trailing others by up to nine points on the large task pool. The authors attribute this partly to output limits and stricter time constraints. The researchers suggest that current benchmarks often fail to account for noise and recommend publishing multiple reruns to verify the actual impact of any agent harness.
