A new study from Google DeepMind and the National University of Singapore, titled 'The Illusion of Stochasticity in LLMs' (arXiv ID 2604.06543), reveals that large language models fail to produce truly random outputs. Researchers presented these findings at the COLM 2026 conference.
The team tested models using Chi-Square and Kolmogorov-Smirnov statistical checks. Results showed that models struggle to follow target distributions, often gravitating toward specific numbers like 7 and 42. P-values were near zero, indicating that native outputs do not match expected patterns.
In game-like scenarios, agents relying on native sampling saw attack success rates rise to between 96.6% and 99.9%, far exceeding the optimal 75% rate. This suggests that LLM-based agents can be highly exploitable in competitive environments due to predictable biases.
The research highlights that while models cannot generate raw randomness, they can accurately transform external random seeds. Advanced models, such as Gemini-3.0-Pro and Qwen3-32B, successfully sampled from desired distributions when provided with random numbers generated by code.
The authors conclude that developers should avoid relying on native LLM randomness for agentic systems. Instead, randomness should be generated via external code and passed to the model to ensure statistical fidelity and security.
