A review queue that made AI output safe to publish

The problem

The agents could write a good response to a customer review. What nobody would accept was those responses going out unread under a client's business name.

So there had to be a person in the loop. The risk in that design is obvious the first time you watch someone use it: if checking a draft takes as long as writing one, the AI has saved nothing and the queue becomes a chore people avoid. The feature is only worth having if review is genuinely faster than authorship.

Designing for the reviewer, not the model

The whole interface follows from one observation. A reviewer is not reading the response — they are comparing it against the review it answers, looking for three specific failure modes: a fact about the business that isn't true, an admission of fault, or an answer to a complaint the customer didn't make.

Everything else was subtraction. No modal to open a draft. No separate edit mode — the response is a text area you can simply type into. No save button; edits are kept as you go, so leaving the page mid-thought costs nothing.

Keyboard first, because volume is the point

People move through dozens of these in a sitting. Reaching for a mouse each time is a tax on the workflow the product depends on, so the queue is fully navigable without one — move between items, approve, skip, jump into the response to edit.

// Approve is optimistic: the row leaves the queue immediately and the next
// draft is focused. If the mutation fails the item returns with an error,
// which is far rarer than the reviewer waiting on a round trip.
function useApprove() {
  const [queue, setQueue] = useOptimistic(items, (current, id: string) =>
    current.filter((item) => item.id !== id),
  );
 
  return async function approve(id: string) {
    setQueue(id);
    await publishResponse(id);
  };
}

Optimistic updates matter more here than on most screens. At one item every few seconds, a 300ms wait per approval is felt as the tool being slow, and a tool that feels slow gets used less.

The states that carried the weight

• Edited drafts are marked. A response a human changed is visibly different from one approved as written — which is also the signal we used to find where the prompts were consistently wrong.
• Nothing publishes on a timer. No auto-approve after N hours. A draft waits until a person releases it, however long that takes.
• Failure is legible. If publishing to the review platform fails, the item returns to the queue saying so, rather than disappearing into a state where nobody can tell whether the customer saw it.

Outcome

The queue is what made the AI features something clients would actually turn on. Giving people approval over the model's work — with the evidence in front of them and the cost of checking pushed as low as it would go — turned the AI from something being done to their brand into something they were operating.

The edit-rate signal turned out to be the most valuable thing the interface produced. Every human correction was a labelled example of the prompt being wrong, and reading a week of them was consistently the fastest way to find what to fix upstream.