Researchers at Sakana AI have introduced PC-ALM, a training method designed as a layer-local alternative to backpropagation. The approach utilizes an augmented Lagrangian to address limitations in predictive coding, allowing for the training of networks with up to 1000 layers.
The method treats every hidden activation as an optimization variable, penalizing mismatches between layers. By incorporating Lagrange multipliers, PC-ALM functions like a PI controller at each layer, where prediction error acts as the proportional term and the multiplier serves as the integral term.
The team demonstrated that in linear networks, PC-ALM converges to the KKT point, with multipliers integrating to exact backpropagation adjoints. The iteration matrix produces damped oscillations, with the parameter alpha controlling frequency.
In tests on MNIST and Fashion-MNIST with residual MLPs, PC-ALM matched backpropagation performance across various widths and depths, while standard predictive coding showed significant drops in deep, narrow configurations. The method also showed improvements on ResNet-18 architectures.
The implementation is available under an MIT license as a JAX reference code. It currently runs on CPUs and has been validated on small image benchmarks, including CIFAR-10 and Tiny ImageNet, with training sessions lasting up to 5 epochs.
