TL;DR: ML CI/CD retains everything standard CI/CD does (lint, tests, build, deploy) and layers on three items, because the model is a function of data, not code alone: validate incoming data (schema, distribution), gate on model quality (a retrained model has to beat the incumbent on a held-out, time-correct eval, not just clear unit tests), and version data + features + model + code together so any run reproduces. Deployment goes shadow then canary because offline metrics never assure online behavior.
How to approach it. Note the shared base in one sentence, then spend your time on what is ML-specific. The framing that scores: in software, behavior comes from code; in ML it comes from code and data, so the pipeline must test, gate, and version the data path as well. The rest follows from that.
A strong answer. Standard CI/CD still applies in full: lint, unit tests on the code, build artifacts, deploy. ML bolts three things on top, plus a staged deployment discipline.
- Data validation as a gate. Before training or serving, check schema, types, null rates, and feature-distribution sanity against expectations. A silent upstream data change (a column that goes all-null, a unit that flips from cents to dollars) is the single most common ML outage, and ordinary tests never see it because the code is untouched. Run the checks in CI on a sample.
- Model evaluation as a release gate. This is the decisive difference. A build that compiles and passes every unit test can still be a worse model. So the pipeline retrains (or loads the candidate), evaluates on a held-out, time-correct eval set, and compares to the current production model. Promote only if it meets or beats the incumbent on the chosen metrics plus fairness and guardrail checks. There is no analog in software CI: passing tests proves correctness, not quality.
- Versioning data + features + model + code together. Reproducibility means pinning the dataset and feature version and the model artifact alongside the code commit (DVC, MLflow, or a Delta version plus a model registry), so any production model can be rebuilt, audited, and a regression traced to exactly what changed.
- Deployment with shadow/canary and rollback. Offline metrics do not predict online behavior (training-serving skew, latency, distribution shift), so promotion runs through shadow then canary with a guardrail that auto-reverts, never a straight cutover.
The mental model: software CI/CD is a subset. ML CI/CD adds a parallel data and model track that runs alongside the code track and joins it at the release gate.
Key takeaways
- The eval-vs-baseline gate is the part with no software equivalent: green tests do not mean a better model.
- A silent data change, not a code bug, is the most common ML outage, so data validation is a first-class CI step.
- Reproducibility requires versioning data and features alongside code and the artifact, not code alone.
- Promotion is staged (shadow then canary) with an automatic guardrail, because offline numbers do not bind online behavior.
What interviewers probe next.
- "Why isn't passing unit tests enough?" Unit tests check code correctness; they cannot tell you the model got worse. You need an eval-vs-baseline gate.
- "What do you version and why?" Data and feature version, model artifact, and code commit, so a result is reproducible and a bad model is attributable to a specific change.
- "How do you test a non-deterministic model in CI?" Assert metrics within tolerance on a fixed eval set rather than exact outputs, and pin seeds where you can.
- "Continuous training?" Automated retraining triggered by schedule or drift, feeding the same eval gate and staged rollout, so the loop is hands-off but still safe.
Common mistakes.
- Describing plain software CI/CD and missing data validation and the model-quality gate.
- Promoting a model because the pipeline "passed" without comparing it to the incumbent.
- Versioning only code, so a production model cannot be reproduced or audited.
- Straight-to-prod deploys with no canary, trusting offline metrics.