Lab: Pipeline build

Assembly week. By the end of this lab, python ask.py "can I return a scratched trivet?" runs the entire machine: ingest-fresh index, rewrite, retrieve, assemble, generate — and prints a cited JSON answer.

Step 1 — Ingestion becomes real

1. 1Refactor into ingest.py: reads corpus/ (split your corpus.txt into one file per article — now it's a real document folder), extracts, cleans, chunks, writes records.jsonl.
2. 2Make it idempotent: per-document content_hash; unchanged docs are no-ops, changed docs replace their chunks. Print the ingestion log line: seen=20 changed=1 replaced=1 failed=0.
3. 3Test it: edit one article (change the restocking fee), re-run, verify exactly that document's chunks changed.

Step 2 — Indexing with a manifest

1. 1Upgrade build_index.py: batch embedding (batches of 64+), the content-hash embedding cache, output to index-vN/ with the manifest from the lesson.
2. 2Add the startup assertion to the query path: manifest's embedding model must match the query embedder's.
3. 3Rebuild twice in a row; the second build's embedding-call count should be ~zero (cache hit). Prove it from the log.

Step 3 — The query pipeline

1. 1Write pipeline.py: answer_question(question, history=[]) → rewrite (only if history non-empty; use a small fast model call) → retrieve top-k via retrieval-v1 → assemble numbered, fenced, budgeted context → call answer() → parse JSON.
2. 2Wire the trace log: one JSONL line per query with every stage's output.
3. 3Run 5 labeled-set questions end to end. For each: JSON parses? citations point at real S-ids? answer actually supported by the cited chunks (read them)?

The rewrite stage has no code yet — here it is. It only fires when there's conversation history (a cold single question needs no rewrite), and it reuses the same messages.create pattern as answer.py. Concrete prompt first:

import anthropic

REWRITE_SYSTEM = """You rewrite a follow-up question into a standalone search
query. Using the conversation history for context, produce ONE self-contained
query that carries all the context the retriever needs — resolve pronouns and
implicit references. Output only the rewritten query, no preamble, no quotes."""


def rewrite(query: str, history: list[dict]) -> str:
    """Turn a conversational turn into a standalone query.
    Returns `query` unchanged when there's no history to resolve against."""
    if not history:
        return query
    turns = "\n".join(f"{h['role']}: {h['content']}" for h in history)
    client = anthropic.Anthropic()
    msg = client.messages.create(
        model="claude-sonnet-5",   # small/fast model is fine; id is swappable
        max_tokens=128,
        system=REWRITE_SYSTEM,
        messages=[{"role": "user",
                   "content": f"History:\n{turns}\n\nFollow-up: {query}"}],
    )
    return msg.content[0].text.strip()

Step 4 — The two-turn test

1. 1Ask 'what's the return window?' then follow with 'and for trade customers?' — passing history. Verify the rewrite produced a standalone query and retrieval found the trade-program article.
2. 2Check the trace: this is the first time you can watch a conversational question travel the whole pipe. Save this trace in LAB-NOTES; it's your demo artifact.