Confidence scoring

Your pipeline now produces answers, citations, and verification flags. Production needs one more output: how much should anyone trust this specific answer? Not as a vibe — as a computed signal that decides whether the answer ships, hedges, or routes to a human. You built routing tables on model self-reported confidence in Prompt Engineering; RAG gives you something better: independent evidence.

The signals you already have

• Retrieval strength: top rerank score, and the gap to the runner-up. Strong-and-decisive beats strong-and-crowded.
• Coverage: did the top chunks come from one coherent source or scraps of five? Answers assembled from fragments fail groundedness more often.
• Verification: all_verified from the citation gate — the strongest single bit you have.
• Model self-report: the answerable flag and any hedging. Useful, but the weakest signal — models are confident by temperament; that's why it gets the smallest weight, not the biggest.

def confidence(trace) -> str:
    """HIGH / MEDIUM / LOW — thresholds calibrated on the eval set."""
    if not trace.answerable or trace.top_score < FLOOR:
        return "LOW"
    strong  = trace.top_score >= STRONG and trace.score_gap >= GAP
    unified = trace.distinct_docs_in_top3 <= 2
    if trace.all_verified and strong and unified:
        return "HIGH"
    if trace.all_verified and (strong or unified):
        return "MEDIUM"
    return "LOW"   # unverified citations never rise above LOW

One wiring note: trace here uses attribute access (trace.all_verified), but the pipeline stages produce dicts — retrieval returns {top_score, score_gap, distinct_docs_in_top3, ...} and verify_citations returns output["all_verified"]. trace is just a small object built from those dicts, so output["all_verified"] becomes trace.all_verified:

from types import SimpleNamespace

# retrieval_info: dict from the retrieval stage (top_score, score_gap, ...)
# verify_output:  dict from verify_citations (all_verified, ...)
# answer_json:    parsed model output (answerable, ...)
trace = SimpleNamespace(**{**retrieval_info, **verify_output, **answer_json})
grade = confidence(trace)   # now trace.all_verified, trace.top_score, etc.

The shape matters more than the exact rules: verification is a hard gate (unverified never exceeds LOW), retrieval evidence does the ranking, and the thresholds come from calibration — run the eval set, bucket answers by score, check the actual correctness rate per bucket. If your 'HIGH' bucket is 98% correct and 'MEDIUM' is 85%, the labels mean something; publish those numbers to your team and revisit them every index rebuild.

Spending confidence

• HIGH → answer with citations, full stop.
• MEDIUM → answer with visible hedging and prominent sources ('Based on the returns policy, it appears…'), or add the sampled groundedness judge inline before showing.
• LOW → don't bluff: not-found path with nearest sources, or human hand-off with the trace attached (the operator card from Prompt Engineering, now fed by real retrieval evidence).