ARIA: automated root-cause analysis for incidents
A self-hosted AIOps platform that correlates metrics, logs and traces to identify which service caused an incident and shorten time to resolution, with no telemetry leaving the country.
Context
During incidents on a microservice platform, many dashboards alert at once and a large part of the response goes to finding the service that actually failed. Commercial APM tools are expensive, and sending government platform logs to a foreign vendor raises data-sovereignty concerns.
What I did
- Built a self-hosted telemetry stack: OpenTelemetry Collector, Prometheus, Loki, Tempo, Grafana and Alertmanager.
- Wrote a Python reasoning engine that correlates metrics, logs and traces and ranks the likely failing service.
- Used a local LLM (Ollama) only to write the incident summary for humans. The diagnosis itself is deterministic and can be audited.
- Scripted the whole lab so it can be rebuilt from scratch: a 20-service reference application, the monitoring stack and fault-injection scenarios.
Key decisions
Result
Simulations show the approach is feasible. I'm now validating its accuracy, then building dashboards for operators, before piloting it with a public institution.
Next steps
Move to dedicated servers with HTTPS end to end, replace the demo workload with a production-like REST service, and use historical incident data to improve the diagnosis.