Researchers from Stanford’s SNAP group, alongside colleagues from Toronto, Penn, and MIT, have released Memorilla. This memory module converts document collections into 16 vectors for frozen large language models, offering an alternative to standard context window usage or retrieval methods.
The training process involves 194,123,776 parameters for the 8B decoder across three stages. The first stage utilizes 1.98 million Wikipedia examples, followed by 1.34 million examples from QA and summarization datasets in the second stage, and final fine-tuning on eight benchmarks. The entire procedure requires 19 hours on four H100 GPUs.
Performance testing on Qwen3-8B showed varying results across benchmarks. On NarrativeQA, the system achieved a 4,212x compression rate, replacing 67,385 document tokens with 16 tokens. In agent memory tasks, eight tokens improved the success rate of a Qwen3-4B agent from 18.6% to 24.2%.
In terms of latency, Memorilla processes 512 documents in 0.74 seconds on a single H100 GPU, compared to 19.9 seconds for full context. Using an optional memory cache on a B200 GPU, response times reach 8.1 milliseconds for 1 million documents and 33 milliseconds for 4 million, with further optimization possible via torch.compile.
The team released checkpoints for Qwen3-8B and the PersonalizationV4 dataset, which includes 149 synthetic users and 200 conversations each. Users can deploy the system via MemoryVLLM, though the initial setup requires downloading approximately 25 GB of model data.
