Retrieval that stopped the model inventing facts

The problem

A response to a customer review is only useful if it is specific. "We're sorry about your experience" is filler; "our kitchen closes at 9pm on Sundays, which is why the order couldn't go through" is an answer.

Specificity is also where a language model becomes dangerous. Asked to be specific with no grounding, it will produce something plausible — an opening time, a policy, a manager's name — and be completely confident about it. Published under a client's business name, an invented fact is worse than a generic reply.

Approach

The fix is not a better prompt. It is making the true facts available at the moment of generation, and making their absence detectable.

Retrieve per location, not per account. A chain has many locations with different hours, staff and policies. Retrieval scoped to the account would happily surface a different branch's opening times — confidently, and wrongly. Scoping is a filter at query time, not a hope about relevance ranking.

def context_for(review: Review, k: int = 5) -> list[Chunk]:
    """Business facts relevant to this review, for this location only."""
    return store.search(
        embedding=embed(review.text),
        # A hard filter, not a ranking hint. Cross-location leakage is the
        # failure that produces a confident, specific, wrong answer.
        filters={"location_id": review.location_id},
        limit=k,
    )

Chunk on meaning, not length. Splitting a policy document every 500 characters routinely cuts a rule in half, and half a rule retrieves as confidently as a whole one. Chunking on document structure — one policy, one FAQ entry, one set of hours per chunk — meant a retrieved chunk was a complete thought.

Keeping it honest

Two things made the difference between a demo and something publishable.

Facts arrive attributed. Retrieved chunks carry their source, so a reviewer in the approval queue can see which document a claim came from. A response asserting something with no supporting chunk is visible as such.

Freshness is part of relevance. Business details change — hours, staff, policies — and a stale chunk is exactly as retrievable as a current one. Content carries an updated timestamp, and superseded material is removed rather than left to compete with its replacement.

Outcome

Responses became specific where the source material supported it and stayed general where it didn't, which is the behaviour you actually want. Cross-location mix-ups — the most damaging error class, because they are fluent and precise and about the wrong shop — were addressed structurally by the scoping filter rather than by asking the model to be careful.

The broader lesson I took from it: retrieval quality is a data problem wearing a model problem's clothes. Almost every bad answer traced back to chunking, scoping, or staleness, and almost none to the generation step.