Students from the HSE and VK AI services workshop are investigating methods to improve linear attention models. The team aims to bring the performance of these models to the level of standard transformers, which typically rely on quadratic computational costs.
The research focuses on the Dense Attention architecture, which utilizes matrix operations without softmax. This approach allows for processing long sequences and maintaining a fixed-size state during autoregressive text generation, effectively replacing the growing KV-cache.
The team hypothesizes that linear models underperform due to suboptimal architectural configurations rather than just the absence of softmax. They are testing whether adjustments to the number of attention heads, positional encoding, and layer schemes can bridge the quality gap.
The researchers are also evaluating the impact of MLM pre-training on linear attention mechanisms. The goal is to optimize these components specifically for linear architectures without reverting to global softmax-based attention methods.
