The research explores the potential for small language models to improve performance without increasing their size. The author is developing a concept called LANN-4, which aims to combine a fixed-size model with external memory to facilitate continuous learning.
The study tested a small model over four rounds of fine-tuning using a 200-task exam. Results showed an initial performance increase of approximately 5 percentage points after the first round, but subsequent rounds failed to produce further gains, indicating a plateau in learning.
The author compared three modes: cumulative fine-tuning, retraining from scratch, and no training. Both cumulative and scratch methods yielded similar results, while the no-training control group showed no improvement, confirming that the initial gain was not due to random noise.
To determine if the model learned the content or simply adapted to the training format, the author proposes a placebo-style test. This involves comparing training on correct task-solution pairs against training on mismatched pairs to isolate the source of the performance boost.
The project emphasizes rigorous methodology, including pre-registering task sets with digital fingerprints and independent verification of results. The author is currently seeking community feedback to identify potential flaws in the experimental design before proceeding with further testing.
