Researchers from Google DeepMind, including Marius De Kuthy Meurers, Aaron Parisi, and Bernd Bohnet, published a paper on October 21, 2026, examining the Gemma-4 4B model. The study focused on the effects of self-rewarded reinforcement learning on moral argumentation.
The training process involved the model evaluating its own arguments regarding moral dilemmas. To ensure the validity of these dilemmas, researchers utilized an SMT solver to verify that the conflicts presented were genuine.
The results showed that the model's consistency improved significantly. The frequency of answer changes due to the reordering of options dropped from 14.3% to 6.6%.
However, other metrics showed little to no improvement. Sycophancy, or the tendency to align with user views, shifted only slightly from 45.3% to 44.0%. Accuracy in moral fables decreased from 56.5% to 53.4%.
The study concludes that while self-rewarded training makes the model more decisive, it does not necessarily improve its moral reasoning or resistance to user influence.
