deployment

Offline metrics gate releases; online metrics reveal what actually happened with users. What shows depth is understanding why offline wins routinely fail to hold up online, and how the two feed each other.

Two models landing at the same accuracy is common, and accuracy was rarely the deciding factor to begin with. The real tiebreakers are the ones a junior candidate leaves out. This is the checklist a staff engineer works through before shipping either one.

Most CV systems break at the seams: a preprocessing mismatch between training and serving, not the model itself. The signal is naming every stage from ingestion through serving and the single consistency invariant that trips teams up. Here is the answer.

Two problems wear one costume here: how you would have caught this before full rollout, and how you debug it after the fact. Cover both and you show senior judgment. This walks through the staged-rollout and root-cause playbook.

When prompts live inside source code, every wording tweak becomes a full deploy and nobody knows which prompt is running. The fix is to treat prompts as managed, versioned configuration. Here is the platform.

Moving a cloud-API POC into a bank or defense network kills your model provider, your judge API, and your vendor dashboards in one stroke. Learn the signed-bundle supply chain, local evaluation loop, and zero-egress observability that make an air-gapped LLM stack actually operable.

The trap is describing ordinary software CI/CD. The signal is the three additions ML brings: data validation, a model-quality gate against a baseline, and versioning data plus model plus code as a unit. Here is what truly differs.

Deploying a model is more than flipping a switch. The signal is knowing what each rollout strategy validates, why shadow is uniquely useful for ML, and keeping rollback a single step away. Here is the comparison and when each fits.

LLMOps is not MLOps with bigger models. What counts is the genuinely new surfaces (prompts as deployable artifacts, eval-driven development, often no training step) and treating prompts with the same version discipline as code.

A bad model in production needs a fast, safe rollback, but ML rollback is harder than code: the model is data and the truth signal lags. What matters is pre-defined criteria plus a previous version kept warm. Here is the answer.

Shadow deployment validates a new model on real traffic without exposing users to it. What matters is knowing outputs run in parallel then get discarded, and how it complements canary and A/B. Here is the answer.

Champion-challenger is how you improve a production model without betting the business on a hunch: the live model versus candidates competing on real metrics. What matters is the ongoing-competition framing and disciplined, pre-committed promotion.

'Which model is in prod right now, and how did it get there?' should be a one-second lookup with an audit trail. If it isn't, you already have a rollback and compliance problem. Here is the promotion pipeline.

A data engineer renames a column and three production models quietly begin scoring on garbage. Features form a shared contract, and altering them demands the same care as a breaking API change. Here is the discipline it takes.

'Works on my machine' is a dependency and environment problem, and for ML it drags in CUDA versions and exact library pins that shift numerical results. Here is how to bring the production environment in line with training.

A green build says nothing about model quality. The signal is an eval harness that runs deterministically in CI, checks against a frozen baseline, and blocks the merge on regressions. Here is how to make it fast, stable, and trusted.

Promotion is the easy part. Governance covers who may ship what, on whose sign-off, with what documented, and how a model gets retired. Here is the control plane an auditor actually asks for.

Manual rollback means minutes of damage while someone wakes up. What matters is defining the trigger signals, thresholds, and guardrails that revert automatically, without flapping on noise. Here is how to make the loop safe.

Saying 'route 5% to the canary' is the easy part. What matters is how you split traffic deterministically, gather enough signal to decide, and ramp on evidence rather than vibes. Here is the mechanics layer.

A platform that no one uses is wasted headcount; a platform built too early is premature. What matters is justifying the investment by leverage and designing a paved path teams adopt willingly. Here is the framing.

A Deployment and a Service will serve a model, but GPUs upend every Kubernetes default: scheduling, probes, autoscaling signals, and rollouts. Here is the setup that survives production, and when KServe earns its complexity.

A failed prod deployment stress-tests judgment under pressure. Interviewers want to see stabilize-first instincts, clean communication, and a root-cause fix that heads off a repeat. Here is the arc.