Kodik IDE was compared against Codex, OpenCode, and DeepSeek Harness using the DeepSWE v1.1 benchmark, which consists of 113 software development tasks across 91 repositories in TypeScript, Go, Python, JavaScript, and Rust. In all four comparisons, Kodik resolved more tasks than the competing agent shells.
When using the GPT-6 Luna model, Kodik resolved 66 tasks compared to 44 by Codex, representing a 50% increase. Despite higher resource consumption per attempt, the cost per successful resolution was $0.172 for Kodik versus $0.228 for Codex, marking a 25% cost reduction.
With the GPT-6.1 Sol model, Kodik resolved 83 tasks against 81 by Codex. The cost per successful resolution was $0.993 for Kodik compared to $1.229 for Codex, which is approximately 19% cheaper. Kodik utilized fewer input tokens but required more tool calls during the process.
Testing with DeepSeek V4.1 Flash 1017 showed Kodik resolving 86 tasks versus 78 by DeepSeek Harness. Kodik completed tasks faster, averaging 17.60 minutes against 20.95 minutes. The cost per successful resolution was $0.393 for Kodik and $0.373 for DeepSeek Harness.
In the comparison with GLM 5.3 Flash, Kodik resolved 73 tasks compared to 67 by OpenCode. The cost per successful resolution was $0.595 for Kodik versus $0.763 for OpenCode, a 22% improvement. Kodik spent more time per successful resolution, averaging 69.27 minutes against 64.33 minutes.
The developers suggest that agent shell quality is particularly critical for less powerful models, as stronger models may compensate for management deficiencies. The study noted that results may be affected by issues in the DeepSWE v1.1 verification infrastructure, which reported problems in at least 23 of 113 tasks.
