Ingestion

Everything so far assumed clean text arriving politely in a file. Production documents arrive as PDFs with three-column layouts, HTML full of nav bars, Word files with tracked changes, and wikis that mix all three. Ingestion — source → clean text → chunks → records — is where RAG projects spend most of their engineering time, and where quality is silently won or lost before a single vector exists.

The ingestion contract

1. 1Extract — per-format parsers (PDF extractors, HTML-to-text, office converters). Test on your documents: PDF extraction quality varies wildly, and tables are where extractors go to die. If a format extracts badly, that's a scoped sub-project, not a footnote.
2. 2Clean — strip boilerplate (headers, footers, nav, cookie banners), normalize whitespace, drop empty sections. Garbage retrieves: a nav bar embedded 200 times becomes 200 chunks of identical noise that match everything weakly.
3. 3Chunk — your Module 2 chunker, per document type. A help article and a contract need different boundary rules; route by type.
4. 4Record — chunk + metadata (lineage fields doing real work now: source_path, content_hash, ingested_at).

Idempotency and the update problem

The design decision that separates toys from systems: what happens when you run ingestion twice? A toy appends duplicates. A system is idempotent — re-ingesting an unchanged document is a no-op, a changed document replaces all of its old chunks (delete by doc_id, insert new), and a deleted document's chunks leave the index. The content_hash you stored per document makes change detection one comparison. Get this wrong and your index slowly fills with stale policy versions that retrieve alongside current ones — the worst kind of wrong answer, the confidently outdated one.

• Never update in place; replace by doc_id. A changed document may chunk into a different number of pieces — surgical updates corrupt positions.
• Keep an ingestion log: per run — docs seen, changed, replaced, failed. When answers go strange, 'what did ingestion do last night?' is the first question.
• Quarantine failures, don't skip silently. A PDF that failed to parse is a visible gap in the knowledge base, not an invisible one.