Avito has developed an internal chaos engineering platform to proactively identify system vulnerabilities in production. The company uses a combination of the open-source tool LitmusChaos and its proprietary orchestrator, Kratos, to simulate infrastructure failures.
While LitmusChaos operates at the pod level, covering approximately 22% of historical incidents, the addition of Kratos allows for complex domain-specific scenarios, such as database master re-elections. This integration has increased incident coverage to 41%.
The team previously utilized an LLM-based pipeline to analyze 977 post-mortem reports (LSRs). The model, which processed data daily, evaluated the potential for reproducing failures via chaos testing and generated automated insights for 408 services.
The platform currently automates three main scenarios: master switching for MongoDB and Redis, network latency simulation, and graceful degradation checks. Master switching for databases accounts for 95.6% of all automated production runs.
The company prioritizes network-related tests, as a single network scenario can cover multiple failure types, including packet loss and latency. The development of these tools aims to prevent the recurrence of past incidents by integrating findings directly into the testing cycle.