Yandex has released YTsaurus Flow as an open-source technology designed for continuous processing of large data streams. The system handles events such as clicks, likes, and purchases in real time as they occur, rather than in batches. This approach eliminates the latency that typically delays data processing by hours.
The technology is built to manage high workloads while ensuring each event is processed exactly once without duplication or loss. It maintains operational stability during failures by redistributing tasks across available servers. YTsaurus Flow is now available under the Apache 2.0 license for commercial use.
Within Yandex, the tool is already utilized by services like Yandex Market for demand analytics and anti-fraud systems for threat detection. In the advertising division, it has reduced data preparation time for model training from over 10 hours to near-zero latency, processing over 100 GB of data and millions of events per second.
Developed by Yandex Infrastructure and Yandex Advertising, the solution operates within the YTsaurus platform. It allows businesses to integrate real-time data processing with existing batch-processing workflows, enabling the simultaneous analysis of historical and incoming information.
