Researchers from KAIST and Naver AI Lab identified that language models process distinct reasoning steps through specific internal numerical patterns. The study categorized eight operations, such as formula recall, deduction, and computation, using models including Qwen2.5-7B, Qwen3-8B, and Gemma4-31B.
The analysis revealed that these reasoning operations are most clearly separable within the middle layers of the models. Internal representations distinguish the type of reasoning step even when the models use identical words, proving that internal states carry information beyond surface-level tokens.
The team found that reasoning steps rely on preceding context, as blocking attention to the previous 30 tokens weakened the internal signal. Furthermore, the identified patterns remained consistent even when the models produced incorrect mathematical results during task execution.
The findings were validated across additional models, including Llama-3-8B, and classifiers successfully transferred to the GPQA-Diamond and MATH-500 datasets. Future research will explore whether these internal signals can be used to monitor or steer model outputs.
