Fixed vs. semantic chunking

Your lab exposed the problem: whole documents are terrible retrieval units — too much noise per hit, and multi-topic questions can't gather pieces from two places. Chunking — how you split documents before embedding — is the highest-leverage design decision in RAG, and the least glamorous. Teams tune prompts for weeks while a bad chunker upstream caps their ceiling.

What a chunk must be

A chunk is the unit of three things at once: embedding (one vector summarizes it — too many topics and the vector is a blurry average), retrieval (it's what search returns), and context (it's what the model reads). Good chunks are one coherent idea, self-contained enough to be understood alone. Every strategy is an attempt to get that automatically.

The two families

• Fixed-size: every N tokens (typically 300–500), regardless of content. Dumb, fast, predictable — and a genuinely strong baseline. Its sin: boundaries fall mid-thought, splitting 'the return window is 30 days' from '…except for holiday orders'.
• Structure-aware (a.k.a. semantic): split where the document splits — headings, sections, paragraphs. Help-center articles, policies, and manuals carry excellent structure; honoring it yields chunks that are naturally one-idea. The usual implementation is recursive splitting: try section boundaries first; any section still too big falls back to paragraphs, then sentences.
• The hybrid everyone converges on: structure-aware first, with a max-size backstop (split oversized sections) and a min-size floor (merge tiny fragments into their neighbor). This is what HarborDocs will use.

Size: the noise-vs-context dial

• Too small (a sentence): precise vectors, but retrieved text lacks its own context — 'the fee is waived' … which fee? — and answers need many chunks to assemble.
• Too big (whole docs): you saw it — diluted vectors, noisy context, wasted tokens.
• The workable band for prose is roughly 200–600 tokens, biased small when questions are pointed ('what's the restocking fee?') and larger when they need surrounding reasoning. It's a starting band, not a law — Module 3's metrics will let you measure your way to the right size instead of guessing.