New research indicates that large language models exhibit a self-preference bias, choosing their own generated responses in approximately 58% of pairwise comparisons. This tendency is observed across major industry models, including OpenAI's GPT, Anthropic's Claude, and Google's Gemini.
The study suggests that between 40% and 96% of this bias may be attributed to actual quality differences in the outputs. Models with lower perplexity, which often score higher on quality benchmarks, demonstrate a stronger inclination toward their own answers. Additionally, AI judges rarely declare ties, preferring to select a winner even when quality differences are minimal.
The self-preference effect appears to intensify as model capability increases, creating a structural conflict of interest when models evaluate their peers. Critics note that data asymmetry and alignment between training data and evaluation criteria may provide inherent advantages to certain providers.
Researchers plan to conduct further studies throughout 2025 and 2026 to distinguish between legitimate quality-based preferences and harmful bias. Proposed mitigation strategies include using diverse panels of models for evaluation or applying statistical corrections to account for known biases in automated systems.
