TL;DR: Pick neither a pure schedule nor a pure trigger. Make a drift detector the main signal (retrain once input or prediction distributions move, or when tracked performance slips), and add a maximum-staleness schedule as a backstop so the model never rots unnoticed. Each retrain has to clear an offline gate and a canary before it takes over from the incumbent.
How to approach it. Recast the question as "what should trigger shipping a new model," and the truthful answer blends triggers, a fallback, and a promotion gate. Split data drift from concept drift up front, since each calls for a different response, and check how quickly labels show up, because label latency governs which signals you can rely on at all.
A strong answer. A schedule-only policy retrains on the calendar regardless of need: wasteful when nothing changed, and dangerously late when the world shifts mid-cycle. A trigger-only policy can go silent if the detector misses slow decay. So build a hybrid with three parts.
- Primary: drift and performance triggers. Monitor input feature distributions and the prediction distribution against a reference window (PSI or KS per feature), and where labels arrive, the live metric (AUC, calibration, business KPI). Distinguish data drift (inputs move, e.g. a new traffic source) from concept drift (the input-to-label relationship changes, e.g. fraud patterns evolve). Concept drift is the one that genuinely demands retraining; data drift sometimes just needs recalibration.
- Fallback: max staleness. A ceiling (say, retrain at least every N weeks) catches slow decay the detector underweights.
- Promotion gate. A retrained model is a candidate, not a release. It must beat the incumbent on a held-out, time-correct eval set, then survive a shadow or canary on live traffic before promotion. Skip this and an automated retrain on a bad-data day silently ships a worse model.
Label latency shapes everything. If ground truth takes weeks (churn, conversions), you cannot trigger on live accuracy quickly, so input-drift detection and proxy metrics (model confidence, prediction-distribution shift) carry more weight.
| Signal | What it means | First response |
|---|---|---|
| Data drift | Inputs moved, label relationship intact | Investigate source; recalibrate often suffices |
| Concept drift | Input-to-label mapping changed | Retrain on recent labeled data |
| Slow decay, no trigger | Detector underweights it | Max-staleness fallback fires |
Key takeaways
- Drift trigger as primary, max-staleness schedule as fallback, never one alone.
- A retrain is a candidate; promotion requires beating the incumbent offline plus a canary.
- Concept drift demands retraining; data drift may only need recalibration.
- Label latency dictates whether you can trust live accuracy or must lean on proxies.
What interviewers probe next.
- "What exactly do you monitor for drift?" Per-feature PSI/KS against a training reference, prediction-score distribution, and when available the realized metric; alert on sustained shifts, not single-batch noise.
- "Concept vs data drift response?" Concept drift means retrain on recent labeled data. Data drift means investigate the source first; recalibration or input fixes may suffice without a full retrain.
- "How do you avoid retraining on a feedback loop?" The model's own outputs influence future data; hold out exploration traffic or use logged-propensity correction so you do not amplify your own bias.
- "Reproducibility?" Version data, features, code, and model artifacts together (DVC, MLflow, Delta) so any production model can be rebuilt and audited.
Common mistakes.
- "Retrain weekly" with no trigger and no promotion gate.
- Triggering on drift but auto-promoting without an offline-beats-incumbent check, so a bad-data day ships a worse model.
- Conflating data drift and concept drift, and retraining when recalibration was the fix.
- Ignoring label delay, then claiming you will trigger on live accuracy that arrives weeks too late.