Scale AI AI Engineer interview questions

Few-shot accuracy can swing on example order by itself. If your prompt is fragile to things that shouldn't matter, the fix is structural, not lucky example-hunting. Here is what genuinely reduces variance.

CoT is not a blanket upgrade. On some tasks it does nothing or backfires, and the reason points to what to try instead. Here is the escalation ladder past 'let's think step by step'.

A model that scores 92% on your eval can still ship a product users hate, because the model is one component in a system. Teams that run only model evals get blindsided. Here is the distinction that matters.

Push PPO hard enough and true quality peaks then declines while the reward keeps climbing. The signal is knowing the Gold-vs-proxy gap, the KL budget that bounds it, and how you actually measure when to stop.

A reward model can only match the quality of its labels, and human preference labels arrive noisy and inconsistent. The signal is measuring inter-annotator agreement and the concrete steps that raise label quality.

Pairwise asks which one wins, pointwise asks how good a single output is. The signal is knowing which is more reliable, how Elo/Bradley-Terry aggregates comparisons, and where each method quietly misleads.

When a benchmark leaks into pretraining, the score measures memorization rather than ability. The signal is the detection methods (n-gram overlap, canaries, perturbation tests) and why fresh held-out evals matter.

A reward model converts pairwise preferences into a scalar signal RLHF can optimize. The signal is the Bradley-Terry loss, the base-model and head choices, and how you validate it before trusting it.

Everyone can name learning rate and epochs. The signal is knowing that LoRA wants a learning rate roughly 10x higher than full fine-tuning, why alpha/r is the only scaling that matters, and which curve tells you your rank is too big.

Your assistant scores 4.6 out of 5 on every individual turn and still loses the user by turn eight. Per-turn averages hide the failures that actually cancel subscriptions, and here is the session-level eval that catches them.

A paraphrased prompt comes back with a training answer, verbatim. This is small-dataset SFT overfitting, not the pretraining memorization problem people confuse it with, and the fix has a strict order of operations.

Four fusion families, and the choice between them is decided by one question most candidates never ask. Naming CLIP and LLaVA is table stakes; the score comes from knowing which one cannot reason and which one eats your context window.

The shift from one model per task to one model adapted many ways rewrote the job description. The interviewer is checking whether you know which half of the work got nearly free, which half did not move at all, and which problems classical ML still owns outright.

Definitions take thirty seconds. The interview is about the four things log-probs buy you in production: a confidence signal, banning tokens, guaranteeing valid output, and classification you can calibrate. Plus the tokenization gotcha that quietly breaks all of them.

The follow-up every interviewer holds in reserve after you say you fine-tuned something. A task-metric win proves almost nothing on its own. Here is the five-part protocol that separates a shipped model from a demo.

Most staleness complaints are not model problems, and the engineer who says so first wins the round. Here is which changes belong in the index, which belong in the weights, and the lifecycle machinery that keeps months of refreshes from eroding the model.

A one-word rewrite moving your accuracy several points is not a prompting problem, it is an uncalibrated decision function. The fix is measuring the model's prior over labels and dividing it out, not another round of tinkering.

Letting a model write your prompts works, right up until the optimizer memorizes your eval set and hands you an unreadable prompt that scores better and generalizes worse. The bounds are the whole answer.

Naming RLHF is not an answer. The frame that scores is outer versus inner alignment, why Goodhart breaks any proxy objective you optimize hard enough, and why a classifier sitting at the boundary can constrain a model's behavior without changing what it is trying to do.

Final-answer accuracy conceals how an agent got there. The signal is trajectory-level metrics: tool-selection and argument accuracy, step efficiency, and matching against reference paths, plus when exact-match is the wrong yardstick.

Screen-driving and browser agents break in ways chat agents never hit: stale DOM, pages that keep shifting, misplaced clicks. What interviewers watch for is the grounding and reliability methods that move a flaky demo toward something dependable.

The agent loop assumes an observation is a short string, and images break that assumption in both directions. The signal is a context policy that keeps pixels out of the transcript and a verification step for the files the agent produces.

A recurring ML-coding screen. The loop itself is easy; what interviewers watch for is k-means++ init, a genuine convergence criterion, and awareness of the failure modes. Here is the compact implementation plus the follow-ups interviewers reliably ask.

Comparing all pairs across a million documents is a trillion comparisons. MinHash estimates Jaccard similarity from a tiny signature, and LSH converts dedup into a near-linear scan. Here is the implementation.

The classic recursive-descent exercise with an applied-AI twist: the JSON your model streams stays truncated mid-token for the whole generation. What matters is a clean strict parser plus a small repair layer, not a second parser. The code follows.

The most job-shaped coding screen in AI, ML, and GenAI engineering: fan out N LLM calls without melting the rate limit or losing the batch to one bad request. What matters is the retry policy, not the async syntax. Below is the version that passes.

The most-asked hands-on LLM exercise: a chat history that outgrows the context window. What gets scored is not the sliding window, it is the region eviction is never allowed to touch. Below is the version that passes.

Asking a model which answer is better is one line of code. Getting a number you would let block a deploy takes position-bias control, a confidence interval, and a human-labeled set the judge is measured against. Here is the harness.

Asking a model 'is this answer hallucinated?' barely beats a coin flip. This is the build: claim extraction, a judge forbidden from using world knowledge, a denominator that excludes hedges, and one model call per claim, which decides where the check can run at all.

A prompt is code, but most teams edit it in a dashboard with no history and then cannot explain last Tuesday's quality drop. This is the small piece of infrastructure that makes prompts reviewable, pinnable, and instantly reversible.

Most RAG projects do not die at the retriever, they die at ingestion. Naive PDF extraction interleaves columns into nonsense, shreds tables, and stamps the footer into all 4,000 chunks. Here is the parser that survives real documents.

A small labeled set is the normal place to start, not an excuse. The strong answer is a ladder of techniques that pull signal from unlabeled data, pretrained models, and the labeling budget. Here it is.

Most real datasets carry wrong labels, and they quietly cap your accuracy. Hand-cleaning all of it does not scale. Here is how to surface the bad labels and train robustly around them.

Most teams run human eval as a vibe check with ten examples and a 1-5 slider. The signal is treating it as a designed experiment: pairwise, blinded, powered, and agreement-measured. Here is the protocol that survives scrutiny.

A model that never looks at the image can score respectably on VQA, which tells you most of what you need to know about the benchmark. Here is what the scores hide, and the one-line ablation that exposes it.

Labeled data is the fuel for ML, and a labeling platform succeeds or fails on quality control. The signal is the workflow plus the quality math: consensus, gold honeypots, inter-annotator agreement, and active learning to spend the budget where it counts.

Batch scoring a billion items is a throughput-and-cost problem, the reverse of low-latency serving. Every choice that harms latency works in your favor here. Here is how to design for dollars-per-million-documents.

When every team evaluates prompts in one-off notebooks, an org ships regressions and argues about vibes. A shared eval platform turns quality into something measurable and comparable. Here is what it must provide.

RLHF and evals can only match the preference data behind them, and that data comes from humans whose quality swings wildly. The platform that yields trustworthy labels is a serious system in its own right. Here is its design.

Models can only be as good as their labels, and noisy annotation quietly caps performance. What interviewers want is a measured quality process, not a bigger collection effort. Here is the answer.

You cannot put a 2 TB dataset in git, and copying it per experiment bankrupts you. What matters is knowing how content-addressed versioning works and when file-level (DVC) versus branch-level (lakeFS) fits. Here is the decision.

Production usage can become your best source of training data, or a self-reinforcing trap. What matters is knowing how to capture clean feedback and how to break the loops that quietly corrupt the model.

Your eval set is clean, well-punctuated, and phrased the way you think about the problem. Your users are not. The move that turns this from a vibe check into an engineering artifact is measuring invariance rather than accuracy, and gating on the number it produces.

Your guardrail reads text. The attack is not in the text. That single sentence is the whole vulnerability, and the red team you build from it looks nothing like the one you already have.

Most AI pilots fail on scoping rather than the model: fuzzy success criteria, the wrong use case, or data that was never ready. For customer-facing and applied AI roles, running a pilot well is the job itself. Here is the playbook.

Scope creep quietly sinks deployments and erodes trust on both sides. Interviewers want to see you defend the timeline without turning into the team that says no to everything. Here is the move.

A failed prod deployment stress-tests judgment under pressure. Interviewers want to see stabilize-first instincts, clean communication, and a root-cause fix that heads off a repeat. Here is the arc.

Executives buy outcomes and care about risk, not your architecture. Interviewers want to see you open with the decision, quantify in their terms, and raise risk honestly. Here is the pattern.

When everything is urgent, the skill is a defensible triage plus honest communication to whoever loses out. Interviewers want a framework and the nerve to say no clearly. Here is the move.

Vague asks are standard in customer-facing AI work. Interviewers look for structured discovery that surfaces the real problem and a measurable first win, not a rush to build. Here is the method.

The opener in half of all Applied AI screens, and most candidates answer it with a definition nobody scores. The distinction that actually earns points is about where you start and what your bottleneck turns out to be.

The classic closer, and most candidates answer it with a horoscope or a list of tool names. The signal is whether you hold an actual view, can justify it from what is already happening, and know what it means for the job.