MLOps & ML Engineering

The rookie reply is 'retrain weekly.' The experienced call is a hybrid: drift-triggered retraining backed by a max-staleness fallback, plus a gate requiring any new model to beat the incumbent. Here is how to think it through.

A feature store is simple to define and simple to botch. The signal is naming the precise bug it prevents (training-serving skew) and the point-in-time correctness that blocks label leakage. Here is the full picture.

The trap is leaping straight to 'add more GPUs.' The signal is profiling first, then reaching for the cheap, no-retrain levers in the correct sequence. Here is the diagnose-then-optimize playbook for p99.

The trap is describing ordinary software CI/CD. The signal is the three additions ML brings: data validation, a model-quality gate against a baseline, and versioning data plus model plus code as a unit. Here is what truly differs.

At thousand-GPU scale, hardware failure is routine rather than rare, and a run that cannot resume burns weeks. The signal is checkpointing strategy, deterministic resume, and holding lost work to a minimum. Here is the fault-tolerant design.

Reproducibility is what divides an ML platform from a heap of notebooks. The signal is capturing the full provenance (data, code, config, metrics) and a registry that governs promotion. Here is what to track and why each piece counts.

Deploying a model is more than flipping a switch. The signal is knowing what each rollout strategy validates, why shadow is uniquely useful for ML, and keeping rollback a single step away. Here is the comparison and when each fits.

Grid search is the wrong answer at scale. The signal is knowing why random beats grid, how Bayesian optimization and early-stopping schemes (Hyperband/ASHA) spend compute wisely, and how to parallelize without stragglers.

GPU spend is typically the largest line item in an AI org, and this question tests whether you reason in utilization rather than just capacity. The signal is wringing out per-unit cost (utilization, right-sizing, spot, quantization) before scaling out.

The most common ML production failure is not a code bug but a quiet upstream data change. The signal is validating at ingestion (schema plus distribution), data contracts with producers, and failing loud rather than training on garbage.

You cannot improve an LLM app you cannot see into, and LLM observability is not service monitoring. What counts is tracing multi-step chains, capturing inputs/outputs/tokens/cost, and online quality signals, not just latency and errors.

Caching is one of the biggest LLM cost levers, but 'cache the response' is naive for a non-deterministic system. What counts is telling prompt (prefix) caching apart from semantic caching and knowing when each is safe.

LLMOps is not MLOps with bigger models. What counts is the genuinely new surfaces (prompts as deployable artifacts, eval-driven development, often no training step) and treating prompts with the same version discipline as code.

Routing each request to the right model is one of the biggest LLM cost/latency levers in production. What counts is matching query difficulty to model capability and knowing when the cascade pattern beats a classifier.

Models decay silently as the world shifts, and 'monitor for drift' is too vague. What counts is the actual statistical methods and separating data drift you can detect without labels from concept drift you often cannot.

A high accuracy number hides slice failures, brittleness, and silent data bugs. What counts is naming the test layers that catch what the headline metric can't. Here is the framework that gets scored highest.

A dependable experiment platform goes well beyond splitting traffic in half. What matters is stable assignment, exposure logging, statistical discipline, and guardrails that hold up against peeking and sample-ratio mismatch. Here is the design.

When a model regresses or an auditor asks 'how was this built,' guessing is a failing answer. What matters is tracking the entire chain so any model can be reproduced and any past decision explained. Here is the chain.

Not every prediction needs a low-latency endpoint, and reaching for one by default wastes money. What matters is matching the serving pattern to the latency and freshness requirement. Here is the decision and when you'd reverse it.

Models can only be as good as their labels, and noisy annotation quietly caps performance. What interviewers want is a measured quality process, not a bigger collection effort. Here is the answer.

A bad model in production needs a fast, safe rollback, but ML rollback is harder than code: the model is data and the truth signal lags. What matters is pre-defined criteria plus a previous version kept warm. Here is the answer.

AI systems get expensive fast, and 'the GPU bill is huge' means nothing you can act on without attribution. What matters is breaking spend down by model, feature, team, and request, then naming the levers. Here is the answer.

Shadow deployment validates a new model on real traffic without exposing users to it. What matters is knowing outputs run in parallel then get discarded, and how it complements canary and A/B. Here is the answer.

ML workflows are multi-step DAGs (ingest, feature, train, eval, deploy), and orchestrators run them reliably. What matters is the DAG/scheduling model plus what's genuinely ML-specific: data deps, versioned artifacts, drift triggers, eval gates.

An ML platform is the internal system that lets many teams build and ship models reliably. What matters is naming the components (data, features, training, registry, serving, monitoring) and arguing why standardizing beats per-team reinvention, plus when not to build.

Champion-challenger is how you improve a production model without betting the business on a hunch: the live model versus candidates competing on real metrics. What matters is the ongoing-competition framing and disciplined, pre-committed promotion.

Monitoring an ML system goes past CPU and latency; the model can silently rot while the dashboard stays green. What matters is the four-layer taxonomy (operational, data, prediction, outcome) and using inputs as leading indicators because labels lag.

Same model, two answers: clean offline, ugly online. The cause is almost always a feature computed differently across the two paths. Here is the diff-based hunt that localizes it to a single column.

'Which model is in prod right now, and how did it get there?' should be a one-second lookup with an audit trail. If it isn't, you already have a rollback and compliance problem. Here is the promotion pipeline.

A fraud model fed a feature an hour behind is half-blind, but recomputing everything in real time wastes money you don't need to spend. Freshness is a per-feature decision on a real cost curve. Here is how to manage it.

Regulators and customers will ask 'Why was I denied?', and 'we retrained since then' will not satisfy them. Reproducing a prediction exactly requires versioning everything. This spells out what 'everything' really covers.

Waiting on labels to measure accuracy means learning a model broke a month after it did. Production monitoring must function before the truth shows up. These are the signals you track instead.

Automated retraining seems like a convenience right up until an auto-retrained model quietly ships worse. The gates are the entire point. Here is how to automate retraining without automating a regression straight into production.

A data engineer renames a column and three production models quietly begin scoring on garbage. Features form a shared contract, and altering them demands the same care as a breaking API change. Here is the discipline it takes.

An alert that fires nonstop gets muted, and a model that fails with no alert is worse still. Good ML alerting is a design problem sharing SRE's principles, with ML-specific twists on top. Here is how to get it right.

'Works on my machine' is a dependency and environment problem, and for ML it drags in CUDA versions and exact library pins that shift numerical results. Here is how to bring the production environment in line with training.

Editing a prompt is a code change that ships without a compiler or a unit test by default, so quality regressions slip out invisibly. Treating prompts and models as testable artifacts is what divides a toy from a product. Here is the harness.

Every team rebuilds a spreadsheet of training runs and then drowns in it. The interviewer wants the platform that ingests runs, params, metrics, and artifacts at high write volume, ties them together by lineage, and makes thousands of experiments comparable, which is a different system from a model registry.

A green build says nothing about model quality. The signal is an eval harness that runs deterministically in CI, checks against a frozen baseline, and blocks the merge on regressions. Here is how to make it fast, stable, and trusted.

Promotion is the easy part. Governance covers who may ship what, on whose sign-off, with what documented, and how a model gets retired. Here is the control plane an auditor actually asks for.

'Just set a seed' misses the point. Reproducibility means pinning the environment, the data, the code, and the hardware-level nondeterminism together. Here is the full checklist and where it springs leaks.

You cannot put a 2 TB dataset in git, and copying it per experiment bankrupts you. What matters is knowing how content-addressed versioning works and when file-level (DVC) versus branch-level (lakeFS) fits. Here is the decision.

Anyone can wire tasks into a DAG. What matters is the patterns that keep it correct under retries and backfills: idempotency, data-aware triggering, and the asset model. Here is what separates a flaky pipeline from a trustworthy one.

Manual rollback means minutes of damage while someone wakes up. What matters is defining the trigger signals, thresholds, and guardrails that revert automatically, without flapping on noise. Here is how to make the loop safe.

Saying 'route 5% to the canary' is the easy part. What matters is how you split traffic deterministically, gather enough signal to decide, and ramp on evidence rather than vibes. Here is the mechanics layer.

A platform that no one uses is wasted headcount; a platform built too early is premature. What matters is justifying the investment by leverage and designing a paved path teams adopt willingly. Here is the framing.

Four drift tests, four different assumptions. The weak answer lists them; the strong one knows which handles high-dimensional embeddings, which requires binning, and which returns a calibrated p-value. Here is how to choose.

Three drifts get casually lumped together, but they differ in detectability and in their fixes. What interviewers want is knowing which you can catch without labels and which one genuinely demands a new model.

Retrain too often and you burn money and risk regressions; too rarely and the model rots. The strong answer is a layered trigger policy with guardrails, not one cron job. Here is how to set the cadence.

You cannot wait for ground truth to tell you the model broke. Prediction-side monitoring catches failures in minutes, not weeks, provided you know which output signals actually move first.

Classic SRE SLOs assume a request is right or wrong. ML predictions are probabilistic and labels lag, so naive uptime SLOs miss the failures that count. Here is how to set SLOs that actually cover model quality.

ML incidents are trickier than service outages: nothing crashed, the model was just wrong. The strong answer covers detection, mitigation, and a blameless postmortem that fixes the system, not the person.

Eyeballing a few outputs does not scale, and a prompt tweak that fixes one case quietly breaks ten. A real LLM eval pipeline pairs a versioned dataset, layered scorers, and a CI gate. Here is the architecture.

Production usage can become your best source of training data, or a self-reinforcing trap. What matters is knowing how to capture clean feedback and how to break the loops that quietly corrupt the model.

A Deployment and a Service will serve a model, but GPUs upend every Kubernetes default: scheduling, probes, autoscaling signals, and rollouts. Here is the setup that survives production, and when KServe earns its complexity.

A provider swap does not fail with a 500. It fails by staying up and quietly getting worse. The signal is knowing what actually changes under you on a model swap, and the shadow-plus-staged-ramp protocol that catches it before your users do.

AI ships behavior, not just code, and behavior is what you need to be able to switch off. The cheapest safety control you will ever build is a toggle that takes effect in seconds, and most teams discover theirs is broken during the incident.

Your eval set is clean, well-punctuated, and phrased the way you think about the problem. Your users are not. The move that turns this from a vibe check into an engineering artifact is measuring invariance rather than accuracy, and gating on the number it produces.

Fairness is a property of the model and the data it meets, so it drifts even when the weights never change. Treating it as a monitoring problem exposes three things a one-off audit never has to solve: noisy small slices, multiple comparisons, and labels that arrive months late.

Migrating models is a project you plan. Being migrated is a Tuesday. When the weights move under a stable API and your prompts were tuned against the old ones, every dashboard stays green while quality quietly rots.