The ASI-Arch system has autonomously designed 105 neural network architectures that outperformed existing models in controlled tests. The project involved 1773 experiments, where the AI proposed structural changes, generated code, executed training, and analyzed performance metrics to refine subsequent iterations.
The research focused on linear attention mechanisms to reduce computational costs. The process began with small-scale training to filter out ineffective designs, followed by testing promising candidates on larger models. The project consumed approximately 20 000 GPU-hours, with half of that time dedicated to the initial research phase.
Out of 1773 initial designs, 1350 passed the first screening. Subsequently, 150 architectures moved to the next stage, where 105 demonstrated superior results on models with 340 million parameters. The five most effective variants were further evaluated on 1.3 billion parameter models across 16 distinct tests.
The top-performing architectures utilized a method of combining multiple information processing paths, dynamically selecting the best approach for specific data. This technique improved performance across various model families, including Mamba2, GLA, and HGRN2, proving the scalability of the discovered principles.
