Module 4Building the pipeline 10 min

Indexing

From records to searchable state: batch embedding, the embedding cache, index versioning, and atomic swaps so users never see a half-built index.

Indexing turns ingestion's records into searchable state: vectors plus the BM25 term index, stored with their metadata. It's conceptually simple — the engineering is in doing it fast, cheap, and without ever serving users a half-built index.

The three habits

Batch your embedding calls. APIs accept dozens-to-hundreds of texts per request; embedding one-at-a-time is 50× slower for identical output. Batch ~100, respect rate limits, retry with backoff on the occasional 429.
Cache by content hash. Before embedding a chunk, check: have we embedded this exact text before? On re-indexing after a small corpus change, the cache turns a full re-embed into a handful of calls. Your content_hash field pays for itself here — same trick our products use for incremental refresh.
Version the index artifact. index-v14/ contains vectors + chunks.json + a manifest: corpus snapshot date, embedding model + version, chunker version, chunk count. When answers change mysteriously, the manifest diff answers 'what changed?' in one look.

manifest.json (inside every index version)python

{
  "index_version": 14,
  "built_at": "2026-07-06T09:12:00Z",
  "embedding_model": "voyage-3.5-lite",
  "chunker_version": "2.1",
  "docs": 20, "chunks": 187,
  "corpus_hash": "9c3fe2…",
  "eval": { "recall_at_5": 0.93, "mrr": 0.81 }   // Module 6 writes this
}

The atomic swap

Rebuilds take minutes; queries arrive continuously. The rule: build the new index completely beside the old one, validate it, then swap a pointer. Never mutate the live index. The validation gate before the swap is your safety net: chunk count within expected range, spot-query sanity checks, and (from Module 6) the eval suite's retrieval metrics meeting their bar. A bad rebuild that never goes live is an incident report you didn't have to write. Bonus: the old index is still on disk, so rollback is re-pointing, not re-building.

The model-swap trap

Changing the embedding model — even a 'minor version bump' — invalidates every stored vector; new-model queries against old-model vectors produce silent nonsense, not errors. The manifest's embedding_model field plus a startup assertion (query model == index model) turns this from a haunting into an error message.

PreviousIngestion Next Query flow