Dashboards that told the team what was breaking

The problem

The platform had grown into a set of services that each knew how they were doing and had no way of saying so. Health was established by asking someone, or by tailing logs until you were reassured.

That works at four services and stops working at a dozen. The specific cost was during incidents: the first twenty minutes went on establishing which part was broken, which is time spent before any repair begins.

What it had to answer

I resisted starting from what data was available and started from the questions people actually asked under pressure:

1. Is anything wrong right now? One screen, answerable in a glance from across a desk.
2. Which service? Every aggregate a way into its components.
3. Since when? Because "did this start with the deploy" is the next question every single time.

Anything that didn't serve one of those three didn't go on the screen. That constraint removed a lot of charts that were interesting and not useful.

Building it

The data came from several places — request metrics, queue depths, job outcomes, database health — with different shapes and update rates. The temptation was to let each panel fetch its own thing on its own schedule, which produces a screen where different tiles are describing different moments in time.

Pre-aggregating on the server was the other decision that mattered. Health questions are asked over hours and days, and shipping raw events to the browser to be summed there meant a screen that got slower exactly as the system got busier — the moment you most need it.

Details that earned their place

• Colour is not the only signal. Status is a shape and a label as well as a colour, because the person reading it at 3am may be on a phone in a dark room, and some of them are colour-blind.
• Absence is rendered. A service that has stopped reporting shows as no data, distinct from healthy. Silence looking like success is how outages get missed.
• Deploy markers on the timeline. Overlaying releases turned "did this start with the deploy" from a question into something you could see.

Outcome

The dashboards replaced the habit of checking health by reading logs. The practical difference during incidents was that the first question — which service — stopped taking twenty minutes, so the work started sooner.

The part I'd carry to any similar project: build from the three questions people ask under pressure, and refuse anything that doesn't answer one of them. Every internal dashboard I've since inherited was suffering from the opposite.