Capstone: Test & red-team
The full test gauntlet: complete eval run, fresh adversarial pass, live conversation drills with a human playing customer, and the honest known-limitations register.
Launch candidates earn the name by surviving testing they didn't design. Four gauntlets, all documented:
- 1The full eval run: entire suite, current build, scorecard vs. your Module 5 baseline — every dimension should hold or improve, and any regression gets explained or fixed. Attach the judge-calibration evidence (a reviewer's first question is 'why should I believe your judge?').
- 2The fresh adversarial pass: 10 new attacks you haven't defended against yet — generated by your AI assistant against your actual system prompt, plus at least two social-engineering conversations that build rapport before striking (multi-turn manipulation is where single-turn attack banks go blind). Expect failures; fix or register them.
- 3Live drills, human customer: recruit one person (colleague, friend) who hasn't seen the build; give them three missions (get a real task done; get a discount by any means; get frustrated on purpose) and no other coaching. Watch without intervening. Live humans find seams scripted attacks never do — usually in turn-taking, tone, and the moments your flows assumed cooperation.
- 4Write the known-limitations register: 5-10 entries, honestly ranked — the attack class that still lands sometimes, the topic where retrieval is thin, the frustration detector's blind spot, the language you don't support. Each with severity, mitigation (guardrail catching it? handoff absorbing it?), and the monitoring that would notice it getting worse. This register is not an admission of failure; it's the difference between a team that knows its system and a team that's about to learn it from customers.
Ready ≠ perfect. Ready = the quality floor holds on the eval suite, no severe known limitation lacks a mitigation, the incident path has names in it, and the off switch is real (whoever owns the live deployment can throw it — verify the fallback UX customers would land on). For this no-code capstone the deliverable is the incident plan with named humans — who can throw the off switch and how — not necessarily you personally pulling a deployment toggle you don't operate; if you do have a live toggle, actually pull it and verify the fallback. Teams that wait for perfect never launch; teams that launch without this list become cautionary transcripts in other people's problem sets.