Sberbank researchers have introduced Pollux, an open-source compact judge model designed to evaluate the quality of other AI systems. The model features 4 billion parameters and is capable of assessing prompt-response pairs, multi-turn dialogues, RAG-based answers, and tool-calling chains in both Russian and English.
The model utilizes the Pollux Benchmark methodology, which includes 35 task groups and 2115 expert-curated prompts. It was trained using the GRPO reinforcement learning algorithm. The system allows users to define custom evaluation criteria, scales, and output formats to suit specific application requirements.
In testing against expert-labeled datasets, Pollux achieved a Spearman correlation coefficient of 0.744, outperforming universal models with up to 1.1 trillion parameters. The solution supports local deployment and provides an API compatible with OpenAI standards.
Developers can integrate Pollux with frameworks such as DeepEval and LightEval. According to Sergey Markov, Director of AI Technology Development at Sberbank, the model provides a cost-effective alternative to using large language models for mass evaluation tasks while maintaining high alignment with human expert assessments.
