Atlassian overhauled its incident detection platform to reduce latency from 40 seconds to under 10 seconds. The new architecture utilizes Apache Kafka, Apache Flink on Kubernetes, and OpenTelemetry to process operational telemetry from over ten cloud products.
The previous Node.js-based system faced issues with high latency, infrastructure dependencies, and rising operational costs, which grew from $120,000 to $230,000 annually. The new pipeline uses a single Flink job to process billions of daily events, significantly improving scalability.
The system incorporates asynchronous enrichment with bulkhead and circuit breaker patterns to prevent bottlenecks. It uses HyperLogLog sketches to track unique users with a 1.5% error margin and implements idempotent sinks to ensure data consistency during restarts.
The AutoHOT decision engine, written in Go, operates in an active-active mode across two regions. It evaluates incident severity based on impact matrices and automates the alerting of on-call teams. The platform successfully increased the detection rate of real incidents to 86% at its peak.
