Module 2Chunking strategies 20 min

Lab: Chunk optimization

Build the real chunker: structure-aware splitting with boundary rules and metadata, re-index the corpus, and beat Module 1 head-to-head.

HarborDocs gets its real ingestion layer. You'll implement the hybrid chunker, attach metadata, re-index, and run the first A/B of the course: chunks vs. whole documents.

Step 1 — The chunker

chunker.pypython

import re, hashlib

MAX_TOKENS, MIN_TOKENS = 500, 60   # ~4 chars/token heuristic is fine here

def chunk_article(doc_id: str, title: str, body: str) -> list[dict]:
    """Structure-aware: split on subsection headings; enforce max/min;
    apply boundary rules; attach metadata; prefix embedded text with
    title + section for context."""
    sections = re.split(r"\n(?=[A-Z][^\n]{0,60}\n)", body)  # heading-ish lines
    chunks = []
    for i, sec in enumerate(merge_small(split_large(sections))):
        heading = first_line_if_heading(sec)
        text = f"{title} — {heading}: {sec}" if heading else f"{title}: {sec}"
        chunks.append({
            "chunk_id": f"{doc_id}#{i}", "doc_id": doc_id, "position": i,
            "title": title, "section": heading or "",
            "category": categorize(title), "status": "current",
            "content_hash": hashlib.sha256(sec.encode()).hexdigest()[:12],
            "text": text,
        })
    return chunks

1Implement the three helpers: split_large (recursive: paragraphs, then sentences), merge_small (fold fragments into the previous chunk), categorize (title keywords → your category enum).
2Encode two boundary rules explicitly: headings stay with their first paragraph; a chunk may not end immediately before a line starting with However/Except/Note.
3Run it over all 20 articles. Eyeball ten random chunks: is each one idea, self-contained, correctly labeled? Fix the regexes until yes — this eyeball pass is real engineering, not busywork.

Try your own versions first — the behavior is described above, and the exact thresholds are the MAX_TOKENS/MIN_TOKENS from chunker.py. Here's one correct implementation of the four helpers (tok() is the same ~4-chars/token estimate the lesson mentions):

chunker_helpers.pypython

import re

# MAX_TOKENS, MIN_TOKENS are defined in chunker.py; mirror them here.
MAX_TOKENS, MIN_TOKENS = 500, 60


def tok(text: str) -> int:
    """~4 chars per token — good enough for boundary decisions."""
    return len(text) // 4


def first_line_if_heading(sec: str) -> str | None:
    """Return the section's first line if it looks like a heading:
    short, no ending period — matches the heading regex in chunk_article."""
    first = sec.strip().splitlines()[0].strip() if sec.strip() else ""
    if first and len(first) <= 60 and not first.endswith("."):
        return first
    return None


def split_large(sections: list[str]) -> list[str]:
    """Any section over MAX_TOKENS is split on paragraph boundaries first,
    then on sentence boundaries — recursively, until each piece fits."""
    out = []
    for sec in sections:
        if tok(sec) <= MAX_TOKENS:
            out.append(sec)
            continue
        parts = re.split(r"\n\s*\n", sec)          # paragraphs
        if len(parts) == 1:
            parts = re.split(r"(?<=[.!?])\s+", sec)  # then sentences
        if len(parts) == 1:
            out.append(sec)                          # atomic; leave it
        else:
            out.extend(split_large(parts))           # recurse
    return out


def merge_small(sections: list[str]) -> list[str]:
    """Fold any fragment below MIN_TOKENS into the previous chunk so no
    chunk is a stray sentence or orphaned heading."""
    out: list[str] = []
    for sec in sections:
        if out and tok(sec) < MIN_TOKENS:
            out[-1] = out[-1].rstrip() + "\n" + sec.strip()
        else:
            out.append(sec)
    return out


_CATEGORIES = {
    "return": "returns", "shipping": "shipping", "warranty": "warranty",
    "care": "care", "assembly": "assembly", "account": "account",
    "order": "account", "gift": "gift-cards", "trade": "trade",
}


def categorize(title: str) -> str:
    """Map a title to a category enum by keyword; default to 'other'."""
    low = title.lower()
    for keyword, category in _CATEGORIES.items():
        if keyword in low:
            return category
    return "other"

The starter kit keeps these four helpers in the same `chunker.py` as chunk_article (shown split into chunker_helpers.py here only to keep the snippet readable) — and they're covered by the offline test suite. One behavior worth noting from the audited kit: categorize returns the first keyword match in dict order, so a title with both words like "Trade Program Returns" resolves to returns (because return precedes trade). That's intended — the kit's tests assert it.

Step 2 — Re-index and A/B against Module 1

1Update build_index.py to embed chunk text and save chunks + vectors (chunks.json + index.npy).
2Write 10 test questions with a note of where the answer lives (article + section). Include your two Module 1 crack-finders and at least one exception-clause question.
3For each question, run top-3 against BOTH indexes (keep Module 1's). Score each hit: does the retrieved text actually contain the answer? Tally: answer-in-top-3 rate, and roughly how much irrelevant text the top hit carries.
4Record both numbers for both indexes in LAB-NOTES.md. Chunking should win visibly on both. If it doesn't, your chunks are probably too small — check the min-merge.

Step 3 — Filter plumbing

1Add a --category flag to ask.py that pre-filters the matrix before top_k (boolean mask on rows).
2Verify: 'how do I care for it?' unfiltered vs. --category care — the filtered version should stop guessing which 'it'.

Problem set 2

In the workbook: a contract PDF, a Slack export, and a product-spec sheet. For each: propose the chunking strategy, the boundary rules that matter, the metadata schema, and the one question format that would still fail — chunking is corpus-specific, and this set proves you can adapt.

PreviousMetadata design Next Recall and precision