The team implemented HolmesGPT, an open-source SRE agent, across six Kubernetes clusters. The agent analyzes alerts by accessing metrics and logs to formulate hypotheses. It operates with read-only permissions, meaning it cannot perform automated repairs.
To integrate HolmesGPT with Alertmanager, the team developed and open-sourced the holmes-alert-bridge service. This tool filters incoming alerts, manages deduplication, and enforces throttling to control costs. It supports delivery via Slack, Telegram, and email.
An investigation typically takes 2–3 minutes and costs between 10 and 30 cents per alert, depending on the model used. In one environment, out of 645 total alerts, only 41 triggered an investigation due to filtering and throttling settings.
The agent utilizes a classic loop to call various tools, such as Prometheus and VictoriaLogs, to gather data. It requires approximately 500 MB of memory to run. The team reports that the generated hypotheses are accurate in 90–95% of cases based on their operational experience.
