Where LLM cost actually goes

The first instinct when an LLM bill gets uncomfortable is to reach for a smaller model. It's the visible lever, and it usually works — right up until quality drops somewhere subtle and you spend a month finding out where.

Every time I've actually got costs down meaningfully, the win came from somewhere else: the model was being sent a great deal of context that had no bearing on the task.

Context stuffing is the default failure

The way most LLM features get built makes this almost inevitable. You start with a prompt. Something goes wrong, so you add the relevant fact to the prompt. It happens again, you add more. Six months later the prompt contains everything anybody thought might help, on every request, because nobody can safely say which parts are load-bearing.

The tell is a prompt that's the same size regardless of how hard the request is. A simple case and a complicated one costing identical amounts means you're paying the complicated price every time.

Tools turn "always" into "when needed"

Exposing context as tools rather than pasting it in inverts the default. The model asks for what the task requires; simple cases cost simple money.

@tool
def account_settings(account_id: str) -> Settings:
    """Brand voice, tone constraints and escalation rules for this account."""
    return settings_store.get(account_id)

This is the same information that used to be in every prompt. The difference is that it's now fetched by the requests that need it — which, when I've measured it, is a minority.

Two things make this work in practice, and both are easy to get wrong:

Scope the tools tightly. A tool that returns "the account's history" will return something enormous. A tool that returns "the last five interactions with this reviewer at this location" returns something useful. The narrower tool is also the one less likely to produce a confidently misattributed answer.

Describe them for a reader who has no other context. The docstring is the entire interface. If a competent new engineer couldn't tell from it when to call the tool, neither can the model.

Measure per component, not per request

The reason cost work stalls is that "we spent this much last month" doesn't tell you what to change.

Attributing spend to prompt components — this instruction block, that retrieved document set, this tool's output — turns the question into an engineering one. Every time I've broken a bill down that way, one or two components have accounted for most of it, and at least one has turned out to be something nobody could justify keeping.

The part that surprises people

Quality tends to go up when you do this, not down.

Long prompts full of marginally relevant context aren't neutral. They dilute the instructions that matter, and they give the model more material to pull an irrelevant detail from. Removing the parts that weren't earning their tokens usually makes the remaining instructions more salient.

That's the argument I'd make for doing cost work before reaching for a cheaper model: it's the only version of this where the two things you care about move in the same direction.

What I'd do first

If a bill is uncomfortable and you haven't looked yet, in order:

1. Attribute spend by prompt component. You cannot fix what you cannot see.
2. Find the context sent unconditionally. Ask what fraction of requests actually use it. This is where the answer usually is.
3. Convert the biggest offender to a tool. Measure quality either side — you're checking a hypothesis, not assuming one.
4. Then, if it's still too high, look at the model.

Reaching for step four first isn't wrong, exactly. It's just the step where you trade something you care about, and the earlier ones usually aren't.