Design & test
The build phase: system prompt, schema, examples, grounding if needed, golden set, one A/B cycle, and the guardrail spec — the full stack, for your case.
Next, build. You've done every step once on the triage assistant; now you'll do them in order, faster, on your own system. Budget three to four hours of focused work. The checklist below is the course in miniature — use it as your build order and your completeness check.
The build order
- 1System prompt v0.1 (M1): sectioned — ROLE / RULES / EDGE CASES / DATA HANDLING / OUTPUT. Rules from your failure-cost table; data-handling line naming your fenced input tag.
- 2Schema (M2): from your brief's v0, with the null policy and (if any field is judgment-heavy) the capped reasoning field first.
- 3Zero-shot baseline (M3 discipline): run your three real samples plus five more you write. Score by hand. Only now decide whether you need examples — and add at most three, targeting observed misses.
- 4Grounding (M4, if your case needs reference documents): fenced, chunk-labeled, closed-book rules, citations with verbatim quotes.
- 5Golden set (M5): 20 cases — ⅓ routine, ⅓ boundary, ⅓ adversarial (include one injection and one impossible-input case). Gold outputs for all. Hold out 3.
- 6One full A/B cycle (M5): hypothesis → pre-registered predictions → run → paired comparison → decision → log line. Reviewers will ask to see this.
- 7Guardrail spec (M6): three-tier validation checklist, retry template, routing table checked against the golden set, operator card, two user-facing messages, log-line fields.
The two traps of the build phase
- Polishing the prompt instead of finishing the system. A 9/10 prompt with no validator loses to a 7/10 prompt with guardrails — in reliability, and in grading. Timebox prompt tuning; the rubric weights the system.
- Golden-set theater. Twenty easy cases that all pass prove nothing. Your set must contain cases you expect to be hard — the grader (and reality) will supply them if you don't.
Act as a hostile QA engineer for my prompt system. Here are my system prompt, schema, and routing table: [paste]. Generate 8 adversarial inputs designed to break it: boundary-straddlers, injection attempts, impossible inputs, format torture. For each, predict what my current design does and whether that's acceptable per my own failure-cost table.
Run its 8 cases for real. Anything that breaks goes in the golden set; anything that breaks badly gets a guardrail. This prompt is your capstone's red team.
Version-freeze everything before moving on: prompt vX.Y, golden set, scores, guardrail spec. The documentation phase documents what IS, not what you're still tweaking. Shipping means stopping.