Freshness, volume & drift
The failures no row-level test can see: data that stops arriving, arrives in the wrong amount, or slowly stops meaning what it used to mean.
Row-level tests check the data that arrived. The nastiest incidents are about data that didn't: the export job that silently stopped Tuesday, the batch that arrived half-sized, the distribution that drifted so slowly no single day looked wrong. These need table-level monitors comparing today against a learned baseline — the second half of any real quality practice.
- Freshness — how stale is the newest data? Define expected cadence per table (orders: daily by 6am), alert when
max(order_date)falls behind. The most common data incident in industry is simply 'the job didn't run and nobody noticed' — freshness monitoring is the cheapest insurance in this course. - Volume — is today's row count plausible? Compare against a trailing baseline (e.g. 7-day average, excluding today). A day at 40% of baseline means a partial load; 230% means a double-send upstream of your idempotent loader. Use a band, not an exact number — real traffic breathes (weekends, promos).
- Distribution drift — are the values still shaped the same? Null rates, category shares, and numeric percentiles, each compared to its baseline. Drift is the slow killer: a payment provider migration moves
statusmix 1% per week for a quarter, every daily check passes, and the quarterly refund number is suddenly indefensible.
Baselines: learned, not guessed
The design pattern for all three: store the daily profile (previous lesson), compute a rolling baseline from it, alert on deviation. A simple, robust scheme — flag when today's value falls outside the baseline mean ± k standard deviations (a z-score) — catches most real incidents with few false alarms. More important than the statistics is the hygiene: exclude the anomalous day from its own baseline, require a minimum history (a 3-day-old table has no 'normal'), and auto-resolve when the metric returns to band.
Why this matters doubly for AI: models and RAG systems consume data silently. A human analyst looking at a half-loaded day says 'huh, that's weird.' A fine-tuning job, a nightly embedding refresh, or an analytics agent says nothing — it just trains on, indexes, or reports the wrong world. Freshness and volume monitors are the smoke detectors for every AI system downstream of the warehouse.
What this module builds by hand — profiles, baselines, z-score monitors, incident lifecycle — is the exact architecture of data observability platforms, including Edova's Vigil: it profiles connected warehouses daily, learns baselines, runs six monitor families (freshness, volume, null spikes, distribution, uniqueness, schema drift), and manages incidents with dedupe and auto-resolve. After this module you won't just be able to use such a tool — you'll know exactly what it's doing and why, which is what lets you calibrate one instead of drowning in its alerts.