ByteDance researchers tested whether LLMs can engineer their own agent harnesses using the HarnessDev framework. The study evaluated six models, including Opus 4.8, GPT-5.5, Gemini 3.1 Pro, DeepSeek V4 Pro, Qwen 3.7 Max, and Seed 2.0 Pro, across five benchmarks totaling 2,207 instances.
The creation process involved building harnesses from a minimal seed, followed by an evolution phase using feedback from 100 SWE-bench Pro and 89 Terminal-Bench 2.1 tasks. Performance was measured by task success and executor token efficiency.
Results showed that code volume did not predict quality, as Gemini achieved high performance with the fewest added lines. Many generated components remained inert, with 124 of 587 writing features identified as dead code and no checkpoint events recorded across 26,679 trajectories.
Generalization proved difficult, as feedback and held-out scores aligned in only 34 of 64 cases. While all five self-runtime creators showed improvement on held-out tasks, progress was not monotonic, and only two of nine final versions were optimal on held-out data.
