Metadata design

A chunk that's just text is an orphan: you can find it but not cite it, filter it, update it, or debug it. Metadata — the structured fields stored alongside each chunk — is what turns a pile of vectors into a maintainable system. Design it at ingestion; retrofitting means re-processing the entire corpus.

The four field families

• Identity: chunk_id, doc_id, position (3rd chunk of doc 7). Non-negotiable — citations (Module 5) and updates (Module 7) both key on these.
• Lineage: source_path, doc_version or content_hash, ingested_at. When a document changes, lineage is how you find and replace its chunks — and how you answer 'which policy version said that?' (the same question you learned to ask in Prompt Engineering's grounding module).
• Filters: the fields queries will constrain on — category (returns/shipping/care…), product_line, audience (customer vs. trade), status (current/archived). Filters come from anticipated questions, not from what's easy to extract.
• Display: title, section_heading, url. What the user sees when you cite — 'Source: Holiday Returns → Extended window' beats 'chunk_147'.

{
  "chunk_id": "returns-holiday#2",
  "doc_id": "returns-holiday",
  "position": 2,
  "title": "Holiday Returns",
  "section": "Extended window",
  "category": "returns",
  "status": "current",
  "content_hash": "a41f…",
  "ingested_at": "2026-07-06",
  "text": "Orders placed Nov 15–Dec 24 may be returned through Jan 31..."
}

Filtering: before, not after

Pre-filtering (search only chunks matching the filter, then rank) beats post-filtering (rank everything, discard non-matching) for a subtle reason: with top-k post-filtering, the k slots can fill up with high-scoring chunks from the wrong category, leaving zero survivors after the filter — a 'no results' that pre-filtering would have avoided. All real vector stores pre-filter; your numpy index can too (mask the matrix rows first). One more design note: inject key metadata into the embedded text itself — embedding "Holiday Returns — Extended window: Orders placed..." rather than the bare paragraph measurably improves matching, because the title context disambiguates pronouns and scope.