Latency optimization

A RAG answer that takes twelve seconds doesn't feel thorough — it feels broken. Before optimizing anything, measure the budget the way users experience it, stage by stage. A typical unoptimized trace:

query rewrite (LLM call) .....  300–800 ms   ← skippable when no history
query embedding ..............   30–80 ms
vector + BM25 search .........    5–50 ms    ← almost never the problem
reranking (API) ..............  100–300 ms
context assembly .............    ~0 ms
generation ................... 1500–6000 ms  ← THE problem
                                 ─────────
                                 2–7 seconds

• Generation dominates. The single biggest lever is output length: a 150-word cited answer generates in a third the time of a 500-word essay. Cap answer length in the schema; users prefer it anyway.
• Stream the answer. Time-to-first-token (~0.5–1s) is what impatience actually measures; tokens flowing feel like progress. Streaming plus a length cap solves 'feels slow' for most products before any architecture changes.
• Skip skippable stages: no history → no rewrite call; identifier regex hit → skip embedding and go keyword; HIGH-confidence cache hit (next lesson) → skip everything.
• Parallelize the independent: vector and BM25 searches run concurrently; so can rewrite-and-embed-original (speculative) if you're fighting for 200 ms.
• Right-size the models: the rewriter needs a small fast model, not your best one; rerankers are already small. Model tiering per stage is free latency.

Budgets, not vibes

Set a per-stage latency budget and alert on p95, not average — retrieval p50 of 40 ms with a p95 of 900 ms means something's wrong (cold cache, rate-limit retries) that averages hide. And keep the escape hatch from your guardrails training: a stage that blows its budget gets skipped or degraded (no rerank beats no answer), logged as such.