Generative AI tools for unit testing offer significant time savings for large organizations. Goldman Sachs utilized the Diffblue Cover tool to generate 3,200 Java tests in one day, a task estimated to take 268 working days manually. However, industry experts express concerns regarding the reliability of these agents.
A primary issue is the tendency of AI to treat bugs as correct behavior, often aligning test cases with existing code flaws. Research from the University of Toronto found that AI models solidified bugs in 3.84% of test cases, a figure that dropped to 2.69% when using docstring descriptions instead of source code as context.
Studies analyzing over 20,000 test sets revealed that AI-generated tests frequently suffer from 'test smells,' such as 'assertion roulette,' where multiple checks lack descriptions. Furthermore, AI agents struggle with exception handling, covering only 7–23% of cases compared to 93% achieved by traditional algorithmic tools like EvoSuite.
Research from ETH Zurich suggests that general-purpose coding agents can outperform specialized test-generation tools. In a benchmark of nearly 2,000 tasks, the SWE-agent successfully resolved 15.9% of cases, while the specialized LIBRO agent achieved 14.1%.
Developers and researchers suggest treating AI-generated tests as drafts requiring human review. Recommended practices include using separate agents for code and test generation to avoid bias, and employing mutation testing to verify if the generated suite can detect intentional bugs.
