The Avito Ads team developed an AI-based pipeline to analyze user feedback. The system replaces manual processing, which became inefficient due to a significant increase in user volume since the service launched in June 2025. The architecture utilizes a multi-agent approach combined with Python scripts to ensure consistent data labeling based on UX research guidelines.
The system breaks down feedback into individual semantic themes rather than treating each comment as a single unit. The pipeline consists of several specialized sub-agents: a segmenter identifies themes, a completeness critic verifies the segments, a classifier applies the taxonomy, and independent critics perform quality checks. Python scripts manage the queue, structure, and data recording.
The team organizes feedback into buckets of up to 50 comments, tracked by a manifest. This modular design prevents context overload and allows for independent error handling. The taxonomy, maintained by UX researchers, defines specific concerns, product areas, and application rules, ensuring that the AI makes decisions based on logic rather than just keyword similarity.
To evaluate performance, the team created a benchmark dataset of 196 manually labeled comments. The primary metric used is Micro F1, which measures the accuracy and completeness of identified concerns. The system currently achieves a Micro F1 score of 83.87 percent. The team continuously updates the benchmark based on findings from the automated analysis.
