Batch vs. streaming
The two ways data moves, why batch is the right default for analytics and AI, and the vocabulary — latency, watermark, backfill — you'll use all course.
Data moves from source systems to the warehouse in one of two rhythms. Batch: on a schedule, in chunks — 'every night at 2am, load yesterday's orders.' Streaming: continuously, event by event — 'each order lands in the warehouse seconds after checkout.' The industry spent a decade acting like streaming was the future and batch was legacy. The verdict is in: batch runs the world, streaming earns its keep in specific seats.
- Batch strengths: simple to reason about, cheap, easy to re-run when something breaks (and something always breaks), natural fit for daily/hourly business questions. A failed batch is a bounded, visible incident; a subtly-broken stream is a slow leak.
- Streaming strengths: the few use cases where minutes genuinely matter — fraud checks at checkout, live inventory, operational dashboards someone watches in real time. Costs more in infrastructure and in engineering care.
- The honest question is not 'how fresh could the data be?' but 'how fresh does the decision need it?' Harbor Lane's revenue reporting, customer resolution, and AI training data are all daily decisions. Batch. Its future 'is this card stolen' check would be streaming — and it would buy that, not build it.
Vocabulary that will follow you through the course
- Latency — time from event happening to event queryable. Batch latency is 'up to one schedule interval plus runtime.'
- Watermark — the marker of how far you've loaded ('orders through 2026-05-30 are in'). Incremental loads live and die by honest watermarks.
- Backfill — re-loading history, because the source corrected data or your transform had a bug. Design for backfill from day one — Module 2's whole lab is making re-runs safe.
- Late-arriving data — the order that happened Tuesday but showed up in Thursday's export. If your pipeline assumes 'each file = exactly that day's events,' late arrivals silently vanish or double-count.
For AI specifically, batch has an underrated virtue: reproducibility. 'The model was trained on the warehouse as of the May 30 load' is a sentence you can audit. Streaming systems make 'what exactly did the model see?' genuinely hard to answer — and the AI Governance course will tell you how much that answer matters.
Start batch, daily. Move a specific flow to hourly when a specific decision needs it. Adopt streaming when someone can name the dollar cost of a 15-minute delay. This ordering is boring and correct — most 'we need real-time' requirements dissolve when you ask what the reader does differently.