Standard large language models operate sequentially, processing a prompt and generating a response before accepting new input. This architecture prevents models from reacting to real-time data, such as camera feeds or user interruptions, until the initial generation cycle concludes.
Georgy Yakushev, a researcher at Yandex Research, is investigating methods to enable interactivity during the inference phase. The research focuses on modifying the KV-cache to allow agents to process new information and adjust plans without requiring a full model restart.
The team explores treating the KV-cache as an active state for agents rather than just a tool for acceleration. This approach aims to help systems coordinate tasks like perception, reasoning, and tool usage, which often operate at different speeds.
