Lab: Baseline a system

You can't operate what you haven't measured. This lab establishes your baseline: the AI feature under management, a first golden set, and its honest current numbers. Everything in the course is measured against this.

Step 1 — Choose your system

Honest framing: no mock is shipped with this course. You bring a working AI feature from a prior course and wrap it behind one entry point — everything the course builds assumes that entry point exists.

• Best: a project you built in a prior course — the triage assistant (Prompt Engineering), HarborDocs (RAG), or the resolution agent (Agentic AI). You know its guts, which makes the ops work concrete.
• Whichever you bring, get it running end to end from one function call — run_feature(input) -> output — before going further. That single entry point is what everything wraps.
• No prior project handy? Stub run_feature with a canned dict so the ops layer is still exercisable (shape below), then swap in a real feature later — the harness, tracer, and gates don't care which is behind the entry point.

def run_feature(input: str) -> dict:
    # Wrap your real prior-course feature here (triage / HarborDocs / agent).
    # No feature yet? Stub it so the ops layer is exercisable, swap in later:
    return {"category": "billing", "urgency": "low", "reply": "..."}

Step 2 — Assemble a starter golden set

1. 1Collect 15 realistic inputs for your feature. Reuse cases from the prior course if you have them; otherwise write them across the archetypes you know (routine, boundary, adversarial, unanswerable).
2. 2Write the expected output (or the pass/fail criteria) for each. This is the same golden-set discipline — the arguments about hard cases are the point.
3. 3Save as golden/v1.jsonl: one JSON object per line, {id, input, expected, notes}. JSONL because you'll append to it all course as production surfaces new failures.

Step 3 — The honest baseline measurement

import json, time
from feature import run_feature   # your one entry point

def baseline(golden_path: str):
    rows = [json.loads(l) for l in open(golden_path)]
    results = []
    for r in rows:
        t0 = time.time()
        out = run_feature(r["input"])          # run it for real
        results.append({"id": r["id"], "input": r["input"],
                        "expected": r["expected"], "output": out,
                        "latency_s": round(time.time() - t0, 2)})
    json.dump(results, open("baseline_results.json", "w"), indent=2)
    print(f"ran {len(results)} cases — now GRADE them by hand")

1. 1Run it. Then grade each output by hand: pass / fail, and a one-word failure reason for the fails. Resist automating the grading yet — Module 2 does that; today you feel the ground truth yourself.
2. 2Compute three numbers: pass rate, median latency, and (from the API response usage) rough tokens-per-call. Write them at the top of BASELINE.md.
3. 3Run the whole set a SECOND time. Compare: how many outputs changed? That number — the non-determinism you just measured directly — is why this course exists.