Lab: Citation design

HarborDocs gets its trust layer. By the end, every answer carries verified citations and a calibrated-ish confidence grade — and you'll have personally tried to sneak a hallucination past the stack.

Step 1 — The verification gate

1. 1Implement verify.py from the lesson (with normalization) and wire it into pipeline.py after generation.
2. 2On verification failure: one retry with the violation named ('quote for S2 not found — copy exact text'), then degrade to not-found-with-sources. Log both events distinctly.
3. 3Test by forcing a failure: temporarily edit a chunk after indexing so the model's likely quote no longer matches. Watch the gate catch it.

Mind the seam: answer() returns msg.content[0].text — a string — but verify_citations(output: dict, ...) needs a parsed dict. Between them sits a json.loads(), and that is the single most common place a real pipeline breaks, because a model occasionally wraps its JSON in prose or emits a stray trailing comma. Handle it explicitly — retry once, then flag — rather than letting an exception escape:

import json


def parse_answer(raw: str, chunks: dict, retry) -> dict:
    """Bridge answer() -> verify_citations(). `retry` re-runs generation
    with a 'return valid JSON only' reminder and returns a fresh string."""
    try:
        output = json.loads(raw)
    except json.JSONDecodeError:
        try:
            output = json.loads(retry())          # one retry, stricter prompt
        except json.JSONDecodeError:
            # Still not JSON — flag, don't crash. Treat as unanswerable.
            return {"answer": "", "citations": [], "answerable": False,
                    "all_verified": False, "error": "model returned non-JSON"}
    return verify_citations(output, chunks)

In the starter kit this bridge lives beside answer() in `answer.py`, where parse_answer(raw) does just the parse (tolerating a \\\json fence or surrounding prose) and hands the dict to verify_citations from verify.py as a separate step — the version above folds retry + verification into one function, which is the same seam drawn tighter. parse_answer is one of the offline-tested functions; the answer()` call around it needs your Anthropic key.

Step 2 — Abstention floor + confidence

1. 1Pull top-score distributions from your trace log for 10 answerable and 10 unanswerable questions ('do you price match?', 'what's the CEO's email?'). Pick FLOOR where the distributions separate.
2. 2Implement confidence.py; wire the LOW path to skip or hedge per the lesson.
3. 3Run the full labeled set; record the confidence distribution. If everything is HIGH, your thresholds are flattery; if nothing is, they're paranoia. Tune to match your hand-sense of the corpus.

Step 3 — Red-team the stack

1. 1Gap-filling attempt: ask five questions near-but-outside the corpus ('do you ship to Canada?' when only domestic shipping is documented). Count how many produce confident answers vs. abstentions.
2. 2Blending attempt: ask about a real topic with a plausible-but-undocumented detail baked in ('is the 15% restocking fee waived for members?' — the fee is real, but no member waiver appears anywhere in the docs). The stack should ground the documented fee and refuse the invented waiver, not negotiate a number the corpus never states.
3. 3Over-synthesis attempt: ask the cross-article combination question from Module 2. Run the groundedness-judge prompt on the answer. SUPPORTED everywhere, or did synthesis creep?
4. 4Every failure becomes a labeled-set case + a defense adjustment. Log which layer caught each attempt — that per-layer tally is your defense-stack report card.