Adversarial testing

You've been attacking all course; this module makes it systematic. Red-teaming is adversarial testing as an engineering practice — not one clever exploit, but comprehensive, repeatable, measured probing of your system's defenses. It's the security equivalent of the eval suite: without it, 'we're secure' is a hope, not a claim.

The red-team mindset, made systematic

Cover the attack surface deliberately, not opportunistically. For each channel and threat in your model, generate attacks across categories:

• Injection — direct and indirect, across every input channel (message, retrieved content, tool results, memory).
• Jailbreaks — the technique families (persona, hypothetical, obfuscation, crescendo) against your app's rules.
• Data extraction — attempts to pull the system prompt, another user's data, secrets, or training data.
• Privilege abuse — trying to make the agent exceed its permissions, take unauthorized actions, reach unscoped data.
• Denial / cost — inputs designed to make the system loop, spend, or hang (unbounded consumption).
• Improper output — getting the model to emit content your app might unsafely execute or render.

The attack suite (your security eval set)

Collect your attacks into a versioned suite you can run on demand — the security-flavored golden set. Each entry: the attack input, the channel, the expected defended behavior (blocked? flagged? harmless-because-of-privilege?), and the OWASP category. Now security is measurable: attack success rate over the suite, tracked over time, run in CI so a change can't silently reopen a hole you closed. This is where LLMOps and security meet — the eval gate now guards safety, not just quality.

• Automate the generation, too: an LLM is a tireless red-teamer — 'generate 20 indirect-injection payloads targeting this agent's refund tool, varied in phrasing and channel.' It finds framings you wouldn't. (Ethically, against your own system.)
• Measure defense in depth: for each landed attack, record which layer stopped it — screening, prompt, privilege, output. An attack that reaches the last layer is a near-miss worth studying.
• Re-run after every change. Security regresses exactly like quality: a prompt tweak reopens an injection, a new tool adds agency. The suite in CI is what catches it.