Getting to 99.95% by making failure visible

The problem

Most outages were discovered by customers. Someone would email support, support would ask engineering, and engineering would start looking. By the time anyone knew there was a problem, it had been happening for a while.

The system was not especially unreliable. It was unobservable — it had no way of telling anyone it was in trouble, so the only detector was a person noticing.

What had to change

Logs that can be searched, not read. Everything moved to structured events with stable names and the identifiers you would actually search by — account, request, service — attached to every line rather than to whichever line the author remembered.

Metrics on the things users feel. Not CPU. Request latency at the percentiles that represent slow users, error rates by endpoint, queue depth and job age. Machine metrics tell you a box is busy; they do not tell you anyone is suffering.

Health that means something. A health check returning 200 because the process is alive tells you nothing. Checks were rewritten to exercise the dependencies a request actually needs, so healthy meant can serve traffic.

Alerting on symptoms

The rule I held to was to alert on what users experience, not on causes.

A cause-based alert — this queue is deep, this service is slow — fires constantly in a healthy system and misses novel failures entirely. A symptom-based alert — error rate above threshold, requests slower than the promise — catches things nobody predicted, which is the category that actually causes outages.

Causes still get dashboards. They just don't wake anyone.

Incident response

Instrumentation only shortens detection. Closing the loop needed process, and very little of it:

• One person owns the incident. Not the fix — the coordination. Without it three people investigate the same thing and nobody talks to support.
• A running log as you go. Written during, not reconstructed after. What we saw, what we tried, what changed.
• A blameless review for anything customer-visible, producing one concrete change. Not five, which never get done. One, assigned, with a date.

The single change per incident is what compounded. Individually small — a missing timeout, a retry without backoff, an alert that should have existed — but a year of them is what moved the number.

Outcome

99.95% uptime, sustained. The shift underneath it was that failures started announcing themselves: detection moved from a customer emailing support to an alert firing, which meant the work began while the problem was small.

What I'd emphasise to anyone starting this: instrument before you need it. The useful comparison is always against last week, and you cannot instrument the past.