LLM-as-judge

Assertions scale for free; human grading doesn't. To evaluate qualitative dimensions on every change, you need the LLM judge — a model scoring outputs against a rubric. You've used this pattern in RAG and Agentic AI; LLMOps demands you use it rigorously, because a miscalibrated judge is worse than no judge: it gives false confidence at scale.

The anchored rubric

'Rate this reply 1–5' measures the judge's mood. A rubric with anchored levels measures the thing you care about:

Score the support reply on FAITHFULNESS (1-5):
  5 = every factual claim is supported by the order data or cited policy
  3 = mostly grounded; one unsupported but harmless nicety
  1 = invents a policy, amount, or order fact
Return JSON: {"score": n, "worst_line": "<quote>", "reason": "<one line>"}
// worst_line + reason force the judge to show its work — and give YOU
// something to spot-check.

Wired up as a harness check, the judge call is a (row, output) -> CheckResult that builds the prompt from the rubric, calls the model, and parses its JSON — with a fail-safe so a non-JSON reply never silently passes:

import json, re
from anthropic import Anthropic
from evalharness import CheckResult

client = Anthropic()          # reads ANTHROPIC_API_KEY from the env
MODEL = "claude-sonnet-5"
RUBRIC = """...the anchored faithfulness rubric above, as a string..."""

def faithfulness_check(row, output) -> CheckResult:
    prompt = f"{RUBRIC}\n\nORDER DATA:\n{row['input']}\n\nREPLY:\n{output}"
    for attempt in range(2):                       # try once, retry once
        msg = client.messages.create(model=MODEL, max_tokens=300,
                                     messages=[{"role": "user", "content": prompt}])
        text = msg.content[0].text
        try:
            verdict = json.loads(re.search(r"\{.*\}", text, re.S).group())
            passed = verdict["score"] >= 4         # pass threshold
            return CheckResult(name="faithfulness", passed=passed,
                               detail=f"score={verdict['score']} :: {verdict['reason']}")
        except (AttributeError, KeyError, json.JSONDecodeError):
            continue                               # retry, then fail safe
    return CheckResult(name="faithfulness", passed=False,
                       detail="judge returned non-JSON twice")

Calibration is not optional

Before trusting a judge, you must prove it agrees with you. This step is skipped constantly and is the difference between a real eval and theater:

1. 1Hand-grade 15–20 outputs on the dimension yourself. These are your ground truth.
2. 2Run the judge on the same outputs. Compare: agreement within 1 point on ≥80% → trust it for screening. Below that → the rubric anchors are too vague; sharpen them with a concrete example of a 5 and a 1, and recalibrate.
3. 3Also check agreement at the pass/fail threshold, not just the ±1 bar: does judge ≥ 4 match human ≥ 4 on each case? A judge that's consistently off by one at the 3/4 boundary sails through the ±1 test while being useless at the exact point your gate makes its decision.
4. 4Re-calibrate whenever the rubric, the model, or the feature changes materially. A judge calibrated six months and two model-versions ago is an unknown.

The judge's failure modes (design around them)

• Self-preference / leniency drift — judges tend to score generously and to like verbose, confident answers. Anchored rubrics and a forced worst_line fight this.
• Position and format bias — when comparing two outputs, judges favor the first, or the longer. Randomize order; compare on substance.
• Correlated blind spots — a judge sharing the generator's misconception will bless the error. For the highest-stakes dimension, a different model as judge, or periodic human audit, is the guard.
• Judges screen; humans decide. Use the judge to rank, flag, and gate at volume; spot-check its extremes; never let it be the final word on a high-stakes release.