Alibaba.com launched CommerceAgentBench, a public toolkit on GitHub designed to evaluate how effectively AI models perform real-world commerce tasks. The benchmark tested 13 AI model families across 107 tasks, including procurement, logistics, and after-sales service. To pass, an agent had to complete the task correctly, such as successfully booking a shipment or publishing a live product listing.
The top-performing model, Claude Opus 5, achieved a success rate of 61.7% using the Accio harness. Performance varied by the underlying harness, with the same model scoring 60.7% on Pi and 56.1% on OpenClaw. Gemini 3 Flash recorded the lowest results, passing 29% of tasks across all three tested environments. The study noted that no single model consistently outperformed others across all categories.
The tests revealed that agents struggle with multi-step processes requiring information retention, such as calculating landed costs, resolving return disputes, and managing multi-leg shipping routes. For instance, agents failed to identify payment anomalies within email threads containing 300 messages. On average, the top model required 63 tool calls and 10 minutes to complete a single task.
The benchmark data was derived from 10 million active small business users, 1.6 million conversations, and 200,000 execution traces. Separately, PYMNTS Intelligence reported that 132 million U.S. adults have utilized AI for retail purchases, with Amazon accounting for 59% of these transactions. Additionally, 46% of merchants identified pricing as the function they are least willing to delegate to AI agents.
