Researchers are exploring methods to generate neural network weights on the fly instead of storing large matrices. This approach aims to reduce storage requirements and memory usage by using a 'recipe'—a generator function or small neural network—to reconstruct weights during execution.
Hypernetworks represent one such approach. In tests with Wide ResNet 40-2 on the CIFAR-10 dataset, the number of parameters dropped from 2.236 million to 0.148 million, though classification error increased from 5.66% to 7.23%.
Dynamic hypernetworks, tested on RNN and LSTM models for Penn Treebank, showed improved performance, reducing error from 1.312 to 1.265 bits per character, although the total parameter count increased from 4.25 million to 4.91 million.
The NeRN approach uses coordinate-based generators to represent CNN weights. For ResNet20 on CIFAR-10, a 0.65 MB predictor achieved 91.68% accuracy, nearly matching the original model's 91.69%. Reducing the predictor size to 0.36 MB lowered accuracy to 90.39%.
Neural Metamorphosis utilizes coordinate generators to adapt network width. In experiments with ResNet20 on CIFAR-10, the model achieved 91.76% accuracy at full width and 89.56% at one-quarter width, demonstrating a 75% reduction in channels.
