A recent poker benchmark by GTO Wizard evaluated thirteen large language models (LLMs) in heads-up no-limit hold'em. The models were tested against an agent playing near Nash equilibrium, which previously defeated human professionals. Results showed that even top-tier models struggle to outperform a bot that simply folds every hand, with the best LLM achieving a result of -16 bb/100.
The performance gap stems from fundamental limitations. LLMs often hallucinate game states, such as misidentifying cards or board textures, which invalidates their strategic analysis. Furthermore, models are trained to provide the single most plausible action, whereas optimal poker strategy requires randomized mixed strategies to remain unexploitable. Models also suffer from results-oriented thinking, incorrectly evaluating decisions based on known outcomes.
Technical comparisons highlight the difference between computational solvers and language models. In 2019, the Pluribus AI defeated elite players using a Monte Carlo CFR algorithm, costing approximately $144 to train. In contrast, running an LLM through the GTO Wizard benchmark cost significantly more while yielding inferior results. Unlike solvers, LLMs cannot perform complex probability calculations or maintain internal game state consistency.
The author suggests a division of labor: using specialized game engines to handle calculations, pot odds, and equity, while delegating the explanation of strategic concepts to LLMs. This approach is implemented in the project 'poker-blindspot.tech'. By providing the model with pre-calculated data and isolating hidden information, the LLM can effectively analyze decisions without being tasked with the math it is prone to failing.
