Stripe has introduced an open-source benchmark called Stripe evals, designed to evaluate the performance of AI coding agents in building functional payment integrations. The project, which became publicly available on November 19, 2026, utilizes 11 reproducible environments categorized into backend tasks, full-stack workflows, and specialized gym problem sets.
The benchmark results indicate that while AI agents achieved an average of 78% of the required criteria, only 29% of test runs successfully met all necessary requirements. The framework operates on a Goose-based agent harness and incorporates Model Context Protocol tools to provide agents with terminal access, browser automation, and documentation lookup capabilities.
Stripe reported that its own plugin improved performance scores by 3 to 8 percentage points across various model and harness combinations. Additionally, the plugin reduced the necessity for web searches by 96% and decreased extraneous test code by approximately 40%. Agents averaged 63 turns to complete a single integration task.
Performance metrics on the leaderboard highlight specific model capabilities, such as Claude Opus 4.5 achieving a 92% average on full-stack tasks and GPT-5.2 reaching 73% on gym sets. Stripe intends to publish updated results periodically to assist businesses in selecting optimal agent configurations based on empirical performance data rather than marketing claims.



