VK engineers implemented a canary deployment system for data snapshots within the VK Ads infrastructure. This approach ensures that data updates, such as configuration changes or machine learning models, are tested on a subset of production instances before full-scale rollout.
The system relies on an existing snapshot orchestrator. Instead of building a new delivery mechanism, the team introduced a snapshot operator that manages the lifecycle of updates. It uses a two-stage routing process: a base type for the canary group and a verified type for the main production environment.
The process begins with the snapshot service creating a file in 30–40 minutes. The canary group receives the update first. The operator then monitors technical and product metrics, comparing them against a control group. If metrics remain within defined thresholds, the snapshot is marked as verified.
The operator is built using a Kubernetes CRD-based model, which allows for state management and fault tolerance. It supports multiple Prometheus-compatible metric sources and uses two analysis windows: one to confirm successful deployment and another to evaluate service performance and quality.
Key metrics for validation include error rates, latency, and specific product indicators like ad delivery volume and CPM across key platforms. If anomalies are detected, the system can halt the rollout, trigger a rebuild, or revert to the previous version, preventing issues from affecting the entire cluster.
