The 1trAIner service, which provides AI-driven training plans and advice, encountered a safety issue regarding medical disclaimers. Although the system prompt included instructions to refer users to doctors in case of pain or injury, the model often ignored this when addressing other user queries.
The developers identified the defect while localizing the product into English and Chinese. Testing revealed that the model failed to provide medical warnings in Russian, English, and Chinese. The team updated the prompt with detailed instructions, but the model's performance remained inconsistent, with evaluation scores rising from 2.78 to 4.33 out of 5.
To ensure compliance, the team implemented a code-based 'guard' that checks for pain markers in user inputs and model responses. If a response lacks a medical referral, the system automatically appends a disclaimer. The guard uses specific regular expressions to handle language-specific boundaries, including a custom class for Cyrillic characters, as standard word boundaries in JavaScript do not support them.
A second-level guard was added to ensure that every suggested activity includes a conditional clause regarding pain. The developers noted that using an LLM as an evaluator is unreliable due to high variance in scoring, so they shifted to a deterministic checklist for quality control. The entire evaluation process cost $1.83.
