The developer created a testing framework for an AI mentor integrated into an educational platform. The system uses 61 test cases across 13 categories to evaluate the AI's pedagogical performance, including its ability to use tools and follow instructions without providing direct solutions to students.
Initial testing revealed that the automated checks were unreliable, often failing to detect truncated responses or allowing solutions that used renamed variables. The developer updated the framework to monitor completion status and refine forbidden patterns, which also helped identify that the production environment had been truncating long responses due to a 1024-token limit.
The evaluation compared DeepSeek V4 Flash, YandexGPT Pro, and Qwen3.6 models. DeepSeek V4 Flash showed the best results in the current configuration. The developer noted that costs are significantly influenced by caching policies and reasoning modes, with YandexGPT being approximately 4.5 times more expensive than Flash for the same workload.
The study highlighted that automated metrics can create a false sense of security. Even with refined tests, the AI models exhibited persistent issues, such as providing unsolicited lectures or failing to follow negative constraints. The developer concluded that automated testing is a tool for prioritizing manual review rather than a substitute for it.
