monitoring

A 90% eval score alongside angry users means your eval set doesn't resemble reality. The fix is to make evaluation follow production, not the reverse. Here is how.

'The model got worse' can mean three very different things with three different fixes. Retrain a model that a pipeline bug actually broke and you just bake in garbage. Here is the order to triage it in.

Bad data quietly corrupts everything downstream (dashboards, models, decisions), and 'it ran without error' is not the same as 'it's correct.' The signal is the categories of checks and failing loud at the boundary.

Production drift monitoring frequently lives in the warehouse, not a fancy tool. Comparing two distributions in SQL with a metric like PSI is a few CTEs. Here is how to compute it.

Two problems wear one costume here: how you would have caught this before full rollout, and how you debug it after the fact. Cover both and you show senior judgment. This walks through the staged-rollout and root-cause playbook.

Models fail silently, so the real question is whether you would notice. What interviewers look for: watching the right layers (operational, data, prediction, outcome) and alerting on drift without drowning in false pages. This covers the system and the metrics that count.

An AI support agent working through millions of chats will occasionally be wrong, rude, or unhelpful, and no one can read them all. Watching quality at that scale is a design problem in itself, with its own failure modes.

The rookie reply is 'retrain weekly.' The experienced call is a hybrid: drift-triggered retraining backed by a max-staleness fallback, plus a gate requiring any new model to beat the incumbent. Here is how to think it through.

The most common ML production failure is not a code bug but a quiet upstream data change. The signal is validating at ingestion (schema plus distribution), data contracts with producers, and failing loud rather than training on garbage.

You cannot improve an LLM app you cannot see into, and LLM observability is not service monitoring. What counts is tracing multi-step chains, capturing inputs/outputs/tokens/cost, and online quality signals, not just latency and errors.

Models decay silently as the world shifts, and 'monitor for drift' is too vague. What counts is the actual statistical methods and separating data drift you can detect without labels from concept drift you often cannot.

A bad model in production needs a fast, safe rollback, but ML rollback is harder than code: the model is data and the truth signal lags. What matters is pre-defined criteria plus a previous version kept warm. Here is the answer.

Champion-challenger is how you improve a production model without betting the business on a hunch: the live model versus candidates competing on real metrics. What matters is the ongoing-competition framing and disciplined, pre-committed promotion.

Monitoring an ML system goes past CPU and latency; the model can silently rot while the dashboard stays green. What matters is the four-layer taxonomy (operational, data, prediction, outcome) and using inputs as leading indicators because labels lag.

Same model, two answers: clean offline, ugly online. The cause is almost always a feature computed differently across the two paths. Here is the diff-based hunt that localizes it to a single column.

Waiting on labels to measure accuracy means learning a model broke a month after it did. Production monitoring must function before the truth shows up. These are the signals you track instead.

An alert that fires nonstop gets muted, and a model that fails with no alert is worse still. Good ML alerting is a design problem sharing SRE's principles, with ML-specific twists on top. Here is how to get it right.

Manual rollback means minutes of damage while someone wakes up. What matters is defining the trigger signals, thresholds, and guardrails that revert automatically, without flapping on noise. Here is how to make the loop safe.

Four drift tests, four different assumptions. The weak answer lists them; the strong one knows which handles high-dimensional embeddings, which requires binning, and which returns a calibrated p-value. Here is how to choose.

Three drifts get casually lumped together, but they differ in detectability and in their fixes. What interviewers want is knowing which you can catch without labels and which one genuinely demands a new model.

Retrain too often and you burn money and risk regressions; too rarely and the model rots. The strong answer is a layered trigger policy with guardrails, not one cron job. Here is how to set the cadence.

You cannot wait for ground truth to tell you the model broke. Prediction-side monitoring catches failures in minutes, not weeks, provided you know which output signals actually move first.

Classic SRE SLOs assume a request is right or wrong. ML predictions are probabilistic and labels lag, so naive uptime SLOs miss the failures that count. Here is how to set SLOs that actually cover model quality.

Production usage can become your best source of training data, or a self-reinforcing trap. What matters is knowing how to capture clean feedback and how to break the loops that quietly corrupt the model.

Fairness is a property of the model and the data it meets, so it drifts even when the weights never change. Treating it as a monitoring problem exposes three things a one-off audit never has to solve: noisy small slices, multiple comparisons, and labels that arrive months late.

AI fails in ways traditional software does not: bias, hallucination, harmful output, silent quality regression. The strong answer is a concrete playbook (detect, contain, communicate) plus a blameless post-mortem that ships systemic fixes. Here is the answer.