Alerting and SLOs for five national platforms
Alert policies, SLOs and error budgets for five production platforms: web, USSD, payments, integration hub and AI API.
Code available on request
5production platforms covered
Context
Five production platforms needed consistent alerting based on user impact, and a shared way for engineering and management to discuss reliability.
What I did
- Designed user-impact alert policies across five production platforms and handed them over to the NOC.
- Built an SLO and error-budget framework that measures reliability from the user's side.
- Set latency alerting for the AI API on time to first token, with streaming endpoints judged separately from synchronous ones.
- Built alert queries that survive service renames and still fire correctly on low-traffic services.
- Diagnosed misleading spikes in long-range charts caused by data aggregation, which had been confusing responders.
- Send weekly performance reports internally and to external partners.
Key decisions
Separate thresholds for streaming AI endpoints over one latency threshold for the whole API
Streaming and synchronous calls have different normal response times, so a single threshold gives false alerts on both.
Result
Alert policies handed over to the NOC. SLO reporting gives engineering and management the same reliability numbers.