In 2023, RAG was the primary method for integrating knowledge into LLMs due to limited context windows of 4–8K tokens. Developers relied on complex pipelines involving chunking, embeddings, vector databases, and rerankers to process documents, which often required months of development and ongoing maintenance.
Modern models now support context windows of 1M+ tokens, theoretically allowing them to process large documents directly. However, research indicates that transformer architectures suffer from a 'U-shaped' attention bias, where models struggle to process information in the middle of a long context, leading to accuracy drops when retrieving multiple facts.
Benchmarks like RULER show that while models perform well on simple 'needle in a haystack' tests, their effective working memory is often only 50–65% of their stated capacity. Furthermore, long conversations suffer from 'context rot,' where accumulated history degrades the model's ability to extract specific information, causing performance to drop from 85% to 65–70% after 15 turns.
Studies comparing architectures on 300-page technical documentation found that RAG maintains a stable 85–90% accuracy throughout sessions because each query remains independent. In contrast, full-context approaches are significantly more expensive and prone to attention dilution as the context fills with irrelevant noise.
To manage long-term interactions, experts recommend context engineering strategies such as summarization, clearing tool outputs, and using sub-agents to process tasks in isolated windows. These methods help maintain performance without overloading the model's primary context.
The author suggests avoiding over-engineering by starting with simpler retrieval methods, such as BM25 keyword search, before scaling to complex RAG stacks. Decisions should be based on actual data requirements rather than defaulting to the most complex infrastructure.
