The Marakuya AI agent framework has completed the OSWorld 2.0 and Terminal-Bench 4.0 benchmarks, which evaluate AI performance in real-world computer environments. These tests measure the combined capability of an LLM and its agentic wrapper to perform complex, multi-step tasks using terminal commands, GUI interactions, and file management.
OSWorld 2.0 consists of 108 scenarios across seven domains, including business, engineering, and creative production. The benchmark limits agents to 500 steps per task. Marakuya, utilizing the DeepSeek-v4.1-flash model, achieved a 20.6% success rate in fully solving tasks, placing it alongside Claude Opus 4.8 and ahead of GPT-5.5 in the official leaderboard metrics.
Terminal-Bench 4.0, updated in August 2026, focuses on command-line interface tasks without GUI access. It currently includes 66 tasks with an eight-hour time limit per assignment. Marakuya recorded a 33.9% resolution rate on this benchmark. The tests are designed to assess how models handle hidden files, debugging, and complex software environments.
The evaluation process for these benchmarks includes safety audits to prevent unauthorized system modifications, such as bypassing sandboxes or altering sudoers files. Marakuya passed these safety checks during its OSWorld 2.0 run. Developers noted that performance was partially constrained by API latency and specific setup limitations, such as the lack of OCR capabilities in the judge model for certain tasks.
