Cost management

RAG costs surprise teams twice: first that indexing is so cheap, then that serving is so expensive. Decompose the bill and both surprises make sense — and become controllable.

The bill, decomposed

• Indexing (one-time-ish): embedding your whole corpus costs less than lunch — embedding models are ~100× cheaper per token than generation models, and the cache means you only ever pay for changed chunks. Never the problem.
• Per-query embedding + retrieval: fractions of a cent. Never the problem.
• Reranking: a real line item at volume (per-candidate pricing) — worth it where it moved your metrics, trimmable via candidate count.
• Generation: the bill. And within it, a subtlety: input tokens dwarf output tokens in RAG, because every query ships k chunks of context. Five 400-token chunks is 2,000 input tokens per question before anyone answers anything.

The levers, in order of return

1. 1Send fewer, better chunks. Every retrieval improvement is a cost cut: if reranking lets k drop from 8 to 4, you just halved the dominant cost while improving answers. Quality and cost point the same direction in RAG — rare and wonderful.
2. 2Prompt caching. Your system prompt and instruction block are identical on every call; model APIs cache repeated prefixes at a fraction of the price. Structure prompts static-first (system + instructions, then variable context) to maximize the cacheable prefix.
3. 3Answer caching. Real traffic is repetitive — the same twenty questions dominate. Cache HIGH-confidence verified answers keyed on normalized query + index version; invalidate on index swap (the version key does it for free). A 30% hit rate is a 30% budget cut and a latency gift.
4. 4Tier the models. Small model for rewriting, mid-tier for routine generation, escalate to your best model only on LOW-confidence retries or flagged categories — the same routing table you already run, now with prices attached.

Then make cost a designed number: compute per-query cost from the trace log (tokens × prices, summed per stage), publish it next to your quality scorecard, and set the alert. 'We answer at $0.011/query, p95 $0.019, alert at 2× daily budget' is an engineering sentence; 'the API bill seems high this month' is not.