Anthropic has updated Claude Code to version 2.1.269, introducing a new evaluation system for plugins. Developers can now define test suites within an evals directory, utilizing prompt files and specific grader configurations to assess plugin performance.
The system supports six types of graders: regex, tool_used, tool_order, and file_exists, which are computed locally, alongside llm and baseline graders that require model calls. Each evaluation run compares performance with and without the plugin loaded to measure its specific contribution.
The tool generates reports in HTML format, which can be published to claude.ai. It also provides a CI gate feature, allowing developers to set a scoring threshold and a maximum cost limit, such as $20, for automated testing processes.
The evaluation process involves multiple agent runs and judge calls, with costs billed directly to the user's API account. Anthropic notes that this system helps identify behavioral issues that standard manifest validation cannot detect.
