Researchers evaluated three open-weight large language models (LLMs) on their ability to independently derive 20 fundamental computer science algorithms. The study, titled 'Can Large Language Models Reinvent Foundational Algorithms?', revealed that none of the models could successfully reconstruct 9 of the tested algorithms when access to prior knowledge was restricted.
The team employed a 'machine unlearning' technique to suppress specific knowledge without full retraining. Models were tasked with writing Python code to solve problems like Dijkstra’s or Euclid’s algorithms. Results showed that when tokens related to algorithm names were blocked, success rates dropped by 19–39%, suggesting a heavy reliance on memorized patterns rather than logical derivation.
Performance improved significantly with external guidance. While models struggled without prompts, general hints helped recover additional algorithms. Detailed, step-by-step instructions occasionally allowed models to outperform their original versions. The study also identified 'reasoning collapse,' where models prematurely ended their search for a solution unless an external verification module provided feedback.
The authors note that the experiment serves as an upper-bound estimate due to the limitations of machine unlearning. The findings highlight a critical gap: LLMs often solve problems by retrieving existing knowledge rather than demonstrating an inherent capacity to independently discover foundational algorithmic principles from scratch.
