Caching

Caching is the highest-leverage cost-and-latency optimization in LLMOps, because production traffic is far more repetitive than it looks — the same questions, the same stable prompts, the same reference documents, over and over. Three kinds, in rising order of power and risk.

Prompt caching (safe, use everywhere)

Your system prompt, instructions, examples, and any fixed reference material are identical on every call and often huge. Model APIs let you cache that repeated prefix so it's processed once and reused at a fraction of the price and latency on subsequent calls. The design move: structure prompts static-first — system + instructions + examples + fixed context at the top (cacheable), the variable user input at the bottom. Free money for zero quality risk; the only work is ordering your prompt to maximize the stable prefix.

Response caching (powerful, needs a key)

• Exact-match: cache the full output keyed on (normalized input + version). Identical request → cached answer, zero cost, instant. A support FAQ where the top 20 questions dominate can see a 30% hit rate, which can cut up to ~30% of model-call cost (assuming similar per-request cost across hits and misses) — plus a latency gift on every hit.
• Key on the version. The cache key must include the prompt/model version, so shipping v8 doesn't serve v7's cached answers. This also gives you free invalidation: a new version has a fresh cache.
• Cache only what's safe to reuse — deterministic-enough, non-personalized responses. Don't cache an answer that embeds a specific user's order details and serve it to someone else.

Semantic caching (tempting, handle with care)

Semantic caching returns a cached answer for a similar (not identical) query, matched by embedding — 'what's your return window?' and 'how long do I have to return something?' hit the same cache entry. Real savings on paraphrase-heavy traffic, but a real hazard: 'can I return this?' and 'can I NOT return this?' are embedding-close and semantically opposite. A too-loose similarity threshold serves confidently wrong cached answers. If you use it, set the threshold conservatively, and evaluate the cache like any other component — a wrong cached answer is still a wrong answer.