Idempotency keys, and the bugs they delete

There is a particular kind of bug report that arrives every few months in any system that receives data from someone else: this customer got two of them. Two confirmation emails. Two charges. Two identical rows an hour apart.

The investigation always finds the same thing. The sender timed out, assumed failure, and retried. We had in fact succeeded — we were just slow to say so.

The retry is not the bug

The instinct is to treat the duplicate as an anomaly: add a check, dedupe on a schedule, ask the sender to stop retrying. All of that is fighting the network.

Timeouts are indistinguishable from failures at the client. A sender who does not retry is a sender who silently loses data whenever a response is slow — strictly worse. Every well-behaved API client retries, which means every server receiving them will be asked to process the same logical operation more than once. That is the normal case, not the exception.

So the question is not how to prevent retries. It is how to make the second one cost nothing.

Let the sender name the operation

An idempotency key is just the sender telling you which operation this is, so you can recognise it if you see it again.

The temptation is to derive the key yourself by hashing the payload. Resist it. A customer legitimately sending the same message twice is a different operation with an identical body, and content hashing collapses the two. The key has to come from the sender, who is the only party that knows their own intent.

For webhooks you usually get one for free — the provider's own event id:

create table raw_events (
  -- The provider's event id, not ours. Their retry carries the same value,
  -- so a replay collides here instead of becoming a second row.
  id           text primary key,
  payload      jsonb not null,
  received_at  timestamptz not null default now(),
  processed_at timestamptz
);

async def ingest(event_id: str, payload: str) -> None:
    await db.execute(
        """
        insert into raw_events (id, payload)
        values ($1, $2)
        on conflict (id) do nothing
        """,
        event_id,
        payload,
    )

The on conflict do nothing is the whole mechanism. A replay is not an error to handle, a branch to test, or a log line to investigate. It is a write that affects no rows.

Do the uniqueness in the database

The version of this that looks equivalent and is not:

# Wrong. Two concurrent retries both pass the check, then both insert.
if not await db.fetchrow("select 1 from raw_events where id = $1", event_id):
    await db.execute("insert into raw_events ...")

Read-then-write has a gap between the read and the write, and concurrent retries live exactly in that gap. Retries are frequently concurrent — a sender that times out at three seconds and retries immediately will often overlap with the request it gave up on.

Only a unique constraint actually enforces uniqueness. Anything in application code is a hint.

Return the original result, not an error

A retry that gets a 409 Conflict has not been handled — it has been rejected, and the client still doesn't know whether the operation happened. Well-behaved idempotency returns the same response the first call produced.

That means storing the response, not just the fact of the write. For an ingest endpoint returning 202 this is trivial. For anything that returns a created resource, keep the resource id alongside the key so the replay can return it.

Separate accepting from processing

The related move, and the one that removes most of the remaining pain: make accepting the request and acting on it two different steps.

Persist the raw payload, acknowledge, and process from the stored record. Now being slow at processing never causes a timeout at the sender, so it never causes a retry, so the retries you do get are genuine network failures rather than self-inflicted ones.

It also means a parser bug becomes recoverable. The payload is on disk in the shape it arrived in, so a fix can replay history instead of apologising for the day you lost.

The category it removes

What I find worth the effort is not any single bug — it's that a whole class of them stops being possible. No duplicate rows. No double sends. No reconciliation job. No "did this actually go through" investigation.

The cost is a primary key you were going to need anyway and an on conflict clause. It is one of the few designs where the correct version is also the shorter one.