Lab: Build the dataset

This lab produces the artifact Module 3 trains on. Budget it honestly — data work is 60% of any adaptation project's calendar, and this lab is deliberately proportioned to teach you that in your bones.

The cleaning pipeline's three load-bearing checks, as adaptable function stubs (not a full implementation — wire your own embedding model and schema in):

# Cleaning-pipeline skeleton — adaptable signatures, not a full implementation.
# The three techniques the lesson names, wired as function stubs.

def near_duplicate_ids(rows, threshold=0.95):
    """Embed each row's text, flag pairs with cosine similarity above the
    threshold as near-duplicates (MinHash is a cheaper alternative at scale).
    Return the ids to drop."""
    ...

def schema_valid(row, schema):
    """JSON-schema-validate one row's output; return True/False. A row with a
    malformed output is an instruction to produce malformed output."""
    ...

def contamination_ids(train_rows, eval_rows, threshold=0.95):
    """Same embedding-similarity method across the split: any train row too
    close to an eval row is a contaminant. Return the train ids to drop."""
    ...

1. 1Write the labeling guide (one page, per the lesson): the triage schema, category definitions with boundary examples, tie-breakers, the [unclear] escape convention. Test it: give the guide + 10 tricky inputs to your AI assistant playing a naive labeler; where it stumbles, the guide is ambiguous — fix before proceeding.
2. 2Generate inputs at curriculum proportions: ~1,200 synthetic-but-realistic support emails via a generation prompt seeded with your domain's shapes — deliberately overweighting the hard families (ambiguous category ~20%, minority classes boosted, adversarial/injection ~8%, multi-issue emails, empty-ish messages). Scrub/placeholder anything resembling real PII by construction. Name the circularity honestly: you're generating training inputs from a frontier model and then distilling that same class of model's labels on them — fine for a learning project, but in production you'd want real traffic as the input source, so the student learns your users' actual distribution rather than inheriting the big model's synthetic-data tells.
3. 3Run the teacher: frontier model + labeling-guide prompt over all inputs, producing schema-valid outputs with the uncertainty flag. Cost note in your log (it's real money and part of the distillation economics story). Adjudicate every teacher-flagged ambiguous case by hand; those labels are yours now, not the teacher's.
4. 4Run the cleaning pipeline (write it as a script), with the techniques named: near-duplicate detection — embed each row and drop pairs above a cosine-similarity threshold (MinHash is the cheaper alternative at scale); JSON-schema validation of every row's output (a malformed output is an instruction to produce malformed output); and a train/eval contamination check using the same embedding-similarity method across the split (including against the Module 1 eval set — on the record). Add the length report and the class-balance report vs. the curriculum plan. Then split: ~1,000 train / 100 validation / the rest folded into test.
5. 5Audit and version: the 75-example stratified read against the guide; compute agreement; if under 90%, do the fix-guide-regenerate loop (most first datasets need one loop — that's the lab working, not failing). Then write MANIFEST.md: dataset v1.0, sources, pipeline version, counts, audit score, teacher model + prompt version, date. Commit everything. You now own a dataset the way Module 3 needs you to.