The Recurrent Looped Transformer (RLT) architecture, proposed by a Princeton researcher, integrates a causal encoder with a recurrent decoder. This design allows the model to maintain decoder states across every token, effectively executing 96 logical blocks per token through parameter reuse.
The decoder state consists of the final output and retained keys and values from the sliding window attention mechanism. A gated merge process combines current encoder representations with previous decoder outputs. The system uses a learned start state and an empty cache for initialization before the beginning-of-sequence token.
The reference configuration employs 48 encoder and 48 decoder layers with shared attention and feed-forward network weights. The architecture supports full-sequence next-token prediction during pretraining and utilizes full backpropagation through time to manage gradient paths through the decoder cache.
For multi-turn serving, the model captures snapshots including the encoder cache, memory, decoder state, and position metadata. These snapshots remain independent of the serving split, allowing for reuse unless weight updates occur or prefix edits force recomputation from earlier checkpoints.
