The developer implemented an AI-based personal diary system where a neural network summarizes user entries and generates follow-up questions. To address the challenge of testing non-deterministic LLM outputs, the author employed an 'LLM-as-a-judge' approach, using a secondary AI to evaluate the primary model's performance based on a specific scoring system.
To create a diverse test dataset, the developer used excerpts from the works of 20 writers from different eras and styles, including Tolstoy, Dostoevsky, Dovlatov, and Tolkien. This approach provided a variety of linguistic patterns, ranging from concise notes to detailed self-reflection, which served as a 'golden standard' for evaluating the system's summarization capabilities.
The testing process involved approximately 100 test runs. The developer utilized a custom Gradle task to automate the evaluation, which is triggered manually before releases rather than on every commit to manage costs and API limits. This method allowed the system to reach a minimum quality score of 4 out of 5 before deployment.
The evaluation criteria for the judge AI ranged from 5 (ideal summary) to 2 (poor summary with distorted meaning or unwanted advice). The author notes that results require careful interpretation, as the judge AI itself may occasionally misinterpret its instructions, necessitating adjustments to the judge's prompt rather than the primary model.
