mlops

Most staleness complaints are not model problems, and the engineer who says so first wins the round. Here is which changes belong in the index, which belong in the weights, and the lifecycle machinery that keeps months of refreshes from eroding the model.

Models fail silently, so the real question is whether you would notice. What interviewers look for: watching the right layers (operational, data, prediction, outcome) and alerting on drift without drowning in false pages. This covers the system and the metrics that count.

The rookie reply is 'retrain weekly.' The experienced call is a hybrid: drift-triggered retraining backed by a max-staleness fallback, plus a gate requiring any new model to beat the incumbent. Here is how to think it through.

A feature store is simple to define and simple to botch. The signal is naming the precise bug it prevents (training-serving skew) and the point-in-time correctness that blocks label leakage. Here is the full picture.

The trap is leaping straight to 'add more GPUs.' The signal is profiling first, then reaching for the cheap, no-retrain levers in the correct sequence. Here is the diagnose-then-optimize playbook for p99.

The trap is describing ordinary software CI/CD. The signal is the three additions ML brings: data validation, a model-quality gate against a baseline, and versioning data plus model plus code as a unit. Here is what truly differs.

At thousand-GPU scale, hardware failure is routine rather than rare, and a run that cannot resume burns weeks. The signal is checkpointing strategy, deterministic resume, and holding lost work to a minimum. Here is the fault-tolerant design.

Reproducibility is what divides an ML platform from a heap of notebooks. The signal is capturing the full provenance (data, code, config, metrics) and a registry that governs promotion. Here is what to track and why each piece counts.

Deploying a model is more than flipping a switch. The signal is knowing what each rollout strategy validates, why shadow is uniquely useful for ML, and keeping rollback a single step away. Here is the comparison and when each fits.

Grid search is the wrong answer at scale. The signal is knowing why random beats grid, how Bayesian optimization and early-stopping schemes (Hyperband/ASHA) spend compute wisely, and how to parallelize without stragglers.

GPU spend is typically the largest line item in an AI org, and this question tests whether you reason in utilization rather than just capacity. The signal is wringing out per-unit cost (utilization, right-sizing, spot, quantization) before scaling out.

The most common ML production failure is not a code bug but a quiet upstream data change. The signal is validating at ingestion (schema plus distribution), data contracts with producers, and failing loud rather than training on garbage.

LLMOps is not MLOps with bigger models. What counts is the genuinely new surfaces (prompts as deployable artifacts, eval-driven development, often no training step) and treating prompts with the same version discipline as code.

Models can only be as good as their labels, and noisy annotation quietly caps performance. What interviewers want is a measured quality process, not a bigger collection effort. Here is the answer.

A bad model in production needs a fast, safe rollback, but ML rollback is harder than code: the model is data and the truth signal lags. What matters is pre-defined criteria plus a previous version kept warm. Here is the answer.

ML workflows are multi-step DAGs (ingest, feature, train, eval, deploy), and orchestrators run them reliably. What matters is the DAG/scheduling model plus what's genuinely ML-specific: data deps, versioned artifacts, drift triggers, eval gates.

An ML platform is the internal system that lets many teams build and ship models reliably. What matters is naming the components (data, features, training, registry, serving, monitoring) and arguing why standardizing beats per-team reinvention, plus when not to build.

Monitoring an ML system goes past CPU and latency; the model can silently rot while the dashboard stays green. What matters is the four-layer taxonomy (operational, data, prediction, outcome) and using inputs as leading indicators because labels lag.

Same model, two answers: clean offline, ugly online. The cause is almost always a feature computed differently across the two paths. Here is the diff-based hunt that localizes it to a single column.

'Which model is in prod right now, and how did it get there?' should be a one-second lookup with an audit trail. If it isn't, you already have a rollback and compliance problem. Here is the promotion pipeline.

A fraud model fed a feature an hour behind is half-blind, but recomputing everything in real time wastes money you don't need to spend. Freshness is a per-feature decision on a real cost curve. Here is how to manage it.

Regulators and customers will ask 'Why was I denied?', and 'we retrained since then' will not satisfy them. Reproducing a prediction exactly requires versioning everything. This spells out what 'everything' really covers.

Waiting on labels to measure accuracy means learning a model broke a month after it did. Production monitoring must function before the truth shows up. These are the signals you track instead.

Automated retraining seems like a convenience right up until an auto-retrained model quietly ships worse. The gates are the entire point. Here is how to automate retraining without automating a regression straight into production.

A data engineer renames a column and three production models quietly begin scoring on garbage. Features form a shared contract, and altering them demands the same care as a breaking API change. Here is the discipline it takes.

An alert that fires nonstop gets muted, and a model that fails with no alert is worse still. Good ML alerting is a design problem sharing SRE's principles, with ML-specific twists on top. Here is how to get it right.

'Works on my machine' is a dependency and environment problem, and for ML it drags in CUDA versions and exact library pins that shift numerical results. Here is how to bring the production environment in line with training.

Every team rebuilds a spreadsheet of training runs and then drowns in it. The interviewer wants the platform that ingests runs, params, metrics, and artifacts at high write volume, ties them together by lineage, and makes thousands of experiments comparable, which is a different system from a model registry.

A green build says nothing about model quality. The signal is an eval harness that runs deterministically in CI, checks against a frozen baseline, and blocks the merge on regressions. Here is how to make it fast, stable, and trusted.

Promotion is the easy part. Governance covers who may ship what, on whose sign-off, with what documented, and how a model gets retired. Here is the control plane an auditor actually asks for.

'Just set a seed' misses the point. Reproducibility means pinning the environment, the data, the code, and the hardware-level nondeterminism together. Here is the full checklist and where it springs leaks.

You cannot put a 2 TB dataset in git, and copying it per experiment bankrupts you. What matters is knowing how content-addressed versioning works and when file-level (DVC) versus branch-level (lakeFS) fits. Here is the decision.

Anyone can wire tasks into a DAG. What matters is the patterns that keep it correct under retries and backfills: idempotency, data-aware triggering, and the asset model. Here is what separates a flaky pipeline from a trustworthy one.

Manual rollback means minutes of damage while someone wakes up. What matters is defining the trigger signals, thresholds, and guardrails that revert automatically, without flapping on noise. Here is how to make the loop safe.

Saying 'route 5% to the canary' is the easy part. What matters is how you split traffic deterministically, gather enough signal to decide, and ramp on evidence rather than vibes. Here is the mechanics layer.

A platform that no one uses is wasted headcount; a platform built too early is premature. What matters is justifying the investment by leverage and designing a paved path teams adopt willingly. Here is the framing.

Four drift tests, four different assumptions. The weak answer lists them; the strong one knows which handles high-dimensional embeddings, which requires binning, and which returns a calibrated p-value. Here is how to choose.

Three drifts get casually lumped together, but they differ in detectability and in their fixes. What interviewers want is knowing which you can catch without labels and which one genuinely demands a new model.

Retrain too often and you burn money and risk regressions; too rarely and the model rots. The strong answer is a layered trigger policy with guardrails, not one cron job. Here is how to set the cadence.

You cannot wait for ground truth to tell you the model broke. Prediction-side monitoring catches failures in minutes, not weeks, provided you know which output signals actually move first.

Classic SRE SLOs assume a request is right or wrong. ML predictions are probabilistic and labels lag, so naive uptime SLOs miss the failures that count. Here is how to set SLOs that actually cover model quality.

ML incidents are trickier than service outages: nothing crashed, the model was just wrong. The strong answer covers detection, mitigation, and a blameless postmortem that fixes the system, not the person.

Eyeballing a few outputs does not scale, and a prompt tweak that fixes one case quietly breaks ten. A real LLM eval pipeline pairs a versioned dataset, layered scorers, and a CI gate. Here is the architecture.

Production usage can become your best source of training data, or a self-reinforcing trap. What matters is knowing how to capture clean feedback and how to break the loops that quietly corrupt the model.

A Deployment and a Service will serve a model, but GPUs upend every Kubernetes default: scheduling, probes, autoscaling signals, and rollouts. Here is the setup that survives production, and when KServe earns its complexity.