The integration of Large Language Models (LLMs) into real-world applications has prompted a discussion on aligning AI outputs with societal norms. New regulatory requirements in Russia, specifically Article 4 of the AI legislation, mandate that fundamental models respect traditional spiritual and moral values, including patriotism, family, and social justice.
To implement these requirements, developers are creating specialized benchmarks. One such tool is RUBIN, developed by Mera. The dataset consists of nearly 800 questions focused on Russian cultural codes, such as slang, folklore, and films. The questions were validated through human surveys, with only those achieving at least 50% consensus among native speakers included in the subset.
However, experts point to significant challenges in creating universal value-based benchmarks. Because LLMs operate on probabilistic mechanisms, they cannot guarantee consistent or reproducible results. Furthermore, values are inherently subjective and culturally diverse, making it difficult to construct a single, logically consistent model that satisfies all social and religious groups within a multi-ethnic state.
The complexity is compounded by the nature of language itself. As noted in translation theory, defining the exact meaning of 'saying the same thing' across different contexts remains problematic. For a multi-lingual country like Russia, this implies that benchmarks would theoretically need to be developed for every language spoken by its various peoples to ensure comprehensive adherence to legal standards.
