Small models in production

A tuned student is a file; production is a service. The decisions between them, sized for a team shipping its first adapted model:

• Hosted serving of your tuned model — several platforms serve custom/tuned open models behind an API (upload adapter or merged weights, get an endpoint). Pros: no GPU ops, autoscaling included, the shortest path from Module 3 to production. Cons: per-token pricing narrows the cost win at very high volume, and your model artifact lives with a vendor (the exit question from the strategy course's rubric applies to your own model now — pleasing symmetry).
• Self-hosted on a standard inference server — the open-source inference stack (vLLM and family) on a rented or owned GPU: maximal cost win at sustained volume, full control, and you've acquired an ops surface (uptime, scaling, updates — the automation course's 'you own a product now' lesson, at infrastructure grade). The honest breakpoint: sustained volume high enough that per-token hosted pricing exceeds a GPU's monthly rent with headroom; below it, hosted wins on total cost of ownership almost every time.
• Quantization — take the discount: quantization = storing the model's weights at lower precision — int8 or int4 instead of 16-bit — to shrink memory and speed inference. The standard 8-bit/4-bit schemes cut memory and cost substantially with typically negligible quality loss at this task size — 'typically' being a word your eval suite exists to check: evaluate the quantized artifact, not just the trained one, because the quantized artifact is what ships. Produce it with bitsandbytes for load-time quantization, or export a served artifact via GGUF/llama.cpp or AWQ. When it passes (it usually does for classification), a 4-bit 8B student (≈4GB of weights, down from ≈16GB at fp16) serves comfortably on modest hardware — which is the whole economics story compounding once more.

The vocabulary for the serving conversation

Four terms make you literate in the deployment meeting: latency (time per response — the student's headline win), throughput (requests per second per GPU — what batching buys, what you provision against peak), cold start (idle-scaled serving pays a spin-up delay on the first request — matters for spiky traffic, solved by keep-warm minimums), and context length (your task's real input sizes — triage emails are short, so the student can run with modest context settings, which buys throughput; know your p95 input length from your own data). You don't need to operate all of this on day one — hosted serving handles most of it — but the team that knows its p95 input length and peak RPS gets a deployment that fits; the team that doesn't gets one that's sized by vibes.