The Real Economics of LLM Tokens

The first invoice is always a surprise. You priced the feature at "a fraction of a cent per thousand tokens," shipped it, and the bill arrived an order of magnitude higher than the spreadsheet said. The sticker price wasn't wrong — the mental model was.

Tokens are not the unit of cost. Behavior is.

A single user action rarely maps to a single model call. It maps to a retrieval, a rerank, a plan, a handful of tool calls, and a synthesis — each one dragging its own context along. The context is the quiet multiplier: every turn re-sends the system prompt, the history, and the retrieved chunks. You pay for that prefix again and again.

Four levers that actually move the bill

1. Context discipline. Most prompts carry passengers — stale history, over-retrieved chunks, verbose system text. Trimming the prefix is often the single biggest win because it compounds across every turn.
2. Caching the prefix. If your provider supports prompt caching, structure the prompt so the stable part is cacheable and the volatile part is small.
3. Right-sizing the model. Routing the easy 80% to a small model and reserving the frontier model for the hard 20% changes the shape of the curve.
4. Failure cost. A retry, a hallucinated tool call, a re-ask — these are invisible in the pricing page and very visible on the invoice.

The thing nobody budgets for

Evaluation. The cheapest token is the one you never send because a good eval told you the smaller model was fine. Spend there first.

---

I write weekly about hiring infrastructure and the economics of building with LLMs. This is a lightly-edited draft — reach out if you want the longer version.