The development team at a marketplace implemented an LLM-based layer to automate customer support reminders. The system was designed to identify whether a conversation is finished or requires further action, replacing manual review by managers.
Initial tests used the Qwen3-30B-A3B model to classify dialogue states. The system was integrated with a monitoring mode to track performance without affecting live operations. The average response time for the classification layer was approximately 0.95 seconds.
The team later introduced a second layer to analyze customer responses, categorizing them into actions such as continuing reminders, stopping, or escalating to a human manager. They evaluated performance using a dataset of 500 dialogues labeled by human support staff.
To improve accuracy, the team implemented a cascade architecture consisting of two judge models and one arbiter. The judges included Qwen3-30B-A3B and gpt-oss-120b, while DeepSeek-V4-Flash served as the arbiter, with GLM-4.5-Air as a backup.
Following these updates, the system achieved an increase in action alignment with human labeling from 74% to 84%. The total number of errors was reduced by 38%, while maintaining a similar average cost per decision.
