Lab: Cut cost

A cost-reduction sprint with a quality gate. You'll measure your feature's per-request cost, apply the three levers, and — the crucial part — prove with your eval harness that you cut cost without cutting quality.

Step 1 — Baseline the cost

1. 1From your traces, compute cost per request (mean and p95), tokens in/out, and calls per request. Record in COST.md next to your current eval pass rate. This paired number — cost AND quality — is what every optimization must respect.
2. 2Identify the biggest driver from the trace data: is it input tokens (fat context), output tokens (verbose answers), model choice, or call count?

Step 2 — Apply the levers

1. 1Prompt thrift: cap output length in the prompt/schema; trim any fat from the system prompt and context. Re-order static-first to maximize a cacheable prefix.
2. 2Response cache: add an exact-match cache keyed on (normalized input + version) for a slice of repeatable inputs. Replay traffic with repeats mixed in; measure the hit rate.
3. 3Routing: if your feature has an easy majority, route those to a smaller model (by heuristic or a cheap classifier), keeping the frontier model for hard cases — or add a cascade that escalates on validation failure.

Step 3 — The gate that makes it honest

1. 1After EACH lever, re-run the full eval harness. Record cost AND pass rate. A lever that cuts cost 30% and drops quality 8% is a trade, not a win — decide explicitly whether the use case allows it.
2. 2For the routing change specifically: run the golden set on the small model alone first, so you know exactly which cases it fails and whether your router sends those to the big model.
3. 3Produce the before/after table in COST.md: cost per request and pass rate at each step. The target: meaningful cost reduction with pass rate within noise of baseline.

Step 4 — The one-sentence result

Write the sentence a professional would report: 'Reduced cost per request from $X to $Y (−Z%) via output caps, a 28% cache hit rate, and routing the routine 60% to a smaller model — golden-set pass rate held at N% (holdout confirms).' That sentence, backed by the table, is the deliverable — and it's the difference between 'I made it cheaper' and 'I made it cheaper and can prove it's still good.'