evaluation

Every customer wants to know how to stop the model inventing things. The weak reply is 'better prompts.' The signal is understanding why models hallucinate and stacking grounding, abstention, and measurement into a system you can defend.

Perplexity is the classic LM metric. What matters is knowing it is just exponentiated cross-entropy, what it intuitively captures, and why low perplexity does not mean a good product.

The classic automatic metrics for generated text. Interviewers want you to pair each with its task and identify the blind spot common to all three. Candidates who pass explain why none of them is the last word on quality.

Plenty of people cite benchmark numbers; almost nobody reads them carefully. What shows depth is explaining what each one measures and why leaderboard figures outrun real-world skill. The question the interviewer is saving is how you would spot contamination.

LLM-as-judge is how you grade open-ended generation at scale, and G-Eval put a formal frame on it. What shows depth is naming the biases that make raw judge scores untrustworthy and demonstrating exactly how you harden them.

Offline metrics gate releases; online metrics reveal what actually happened with users. What shows depth is understanding why offline wins routinely fail to hold up online, and how the two feed each other.

'Eval is the new system design' for LLM apps, and starting with zero data is the real test. What shows depth is bootstrapping a golden set, choosing task-specific metrics, and running evals continuously as a regression gate.

Preventing hallucinations is one job; catching the ones that slip through at runtime is a separate one. The signal is the detection toolkit, why each signal is imperfect, and how you merge them into an action.

Hand-tuned prompts break on the next model upgrade and fail to compose across pipeline stages. Frameworks like DSPy compile prompts against a metric instead. The signal is treating prompting as an optimized program. Here is the answer.

A model that claims a million-token window may not actually exploit it. The signal is knowing needle-in-a-haystack, where it fails, and how to probe genuine multi-fact reasoning across the entire context. Here is the answer.

Alignment that piles on refusals and politeness often quietly erodes reasoning and coding. The alignment tax is real and measurable. Here is how to hold onto the safety gains without paying for them in capability.

Push any proxy hard enough and the model discovers the exploit, not the goal. Sycophancy, padding, and fake citations are reward hacking. Here is why it happens and the controls that actually hold.

Few-shot accuracy can swing on example order by itself. If your prompt is fragile to things that shouldn't matter, the fix is structural, not lucky example-hunting. Here is what genuinely reduces variance.

CoT is not a blanket upgrade. On some tasks it does nothing or backfires, and the reason points to what to try instead. Here is the escalation ladder past 'let's think step by step'.

A model that scores 92% on your eval can still ship a product users hate, because the model is one component in a system. Teams that run only model evals get blindsided. Here is the distinction that matters.

For open-ended generation there's no ground-truth string to match, so accuracy is meaningless. The field relies on a layered mix of automatic, model-based, and human metrics. Here is how to assemble a credible eval.

Pairwise asks which one wins, pointwise asks how good a single output is. The signal is knowing which is more reliable, how Elo/Bradley-Terry aggregates comparisons, and where each method quietly misleads.

When a benchmark leaks into pretraining, the score measures memorization rather than ability. The signal is the detection methods (n-gram overlap, canaries, perturbation tests) and why fresh held-out evals matter.

The same model can land ten points apart on MMLU depending on prompt format and scoring method. The signal is knowing the knobs (log-prob vs generation, few-shot, normalization) that make evals reproducible.

Candidates who cite one leaderboard number fail the follow-up. The interviewer wants you to name what SWE-bench measures, what it cannot (long-horizon multi-file refactors, real tool use, business logic), and which complementary benchmarks close those gaps.

Your assistant scores 4.6 out of 5 on every individual turn and still loses the user by turn eight. Per-turn averages hide the failures that actually cancel subscriptions, and here is the session-level eval that catches them.

The shift from one model per task to one model adapted many ways rewrote the job description. The interviewer is checking whether you know which half of the work got nearly free, which half did not move at all, and which problems classical ML still owns outright.

The follow-up every interviewer holds in reserve after you say you fine-tuned something. A task-metric win proves almost nothing on its own. Here is the five-part protocol that separates a shipped model from a demo.

The staged recipe is the easy half. The half that sinks real projects is data licensing, an evaluation bar that needs actual domain experts, and knowing which parts of the problem fine-tuning cannot solve at all.

Summarization is the one task where the model was handed everything it needed and still made something up. The signal is separating intrinsic from extrinsic hallucination, knowing why ROUGE cannot see either, and gating on entailment before you ship.

A one-word rewrite moving your accuracy several points is not a prompting problem, it is an uncalibrated decision function. The fix is measuring the model's prior over labels and dividing it out, not another round of tinkering.

Three different systems are failing at once (the tokenizer, the model, and your pipeline) and most candidates blur them into one. Separating them tells you which fix buys the most, and the highest-leverage one is not the prompt.

The most repeated trick in prompt engineering, and the evidence for it is much thinner than the folklore. A screening question for whether you measure your prompts or inherit them from a thread of screenshots.

One of the most common AI, ML, and GenAI design rounds. Sketching embed-retrieve-generate is easy; the signal lives in chunking, hybrid retrieval, the rerank/latency tradeoff, and how you prove it works. Here is the structure that scores.

Agents get reached for too often. A strong answer pushes past the hype: most tasks are better served by a single structured call, and agents pay off only under particular conditions. Here is when one is worth it and how to stop it from compounding errors.

Most RAG systems ship without genuine evaluation, which is why most of them quietly degrade. The signal is breaking evaluation into retrieval and generation, and scoring faithfulness apart from relevance. Here is the framework that makes RAG improvements measurable.

Reflection is the pattern in which an agent critiques and revises its own work. The signal is knowing when it truly helps (external feedback grounds the critique) versus when self-critique on its own is theater.

A model that leads MTEB can still stumble on your jargon-heavy corpus, and most teams reach straight for the costly fix. There is a ladder, and the cheap rungs are the ones people skip. Here is the order to climb it.

When the right answer isn't even in the top-50, the generator can't rescue you. Low recall has a short list of usual suspects. Here is the order to check them so you fix the cause, not a symptom.

More chunks means higher recall but also more noise, more cost, and more lost-in-the-middle. The right k is an empirical tradeoff, not a default of 5. Here is how to find it.

When a chain of LLM calls and tools yields a wrong final answer, 'the model was bad' is not a diagnosis. You need visibility into every step. Here is the tracing layer that turns a black box into something debuggable.

A 90% eval score alongside angry users means your eval set doesn't resemble reality. The fix is to make evaluation follow production, not the reverse. Here is how.

Six stages, six isolating experiments, six metrics. The candidates who score do not guess at knobs, they bisect the pipeline, and they instrument the rungs nobody else does: ranking, context assembly, generation.

A dense encoder given 'refund policy' has almost nothing to work with. The fix is counterintuitive and it is not a better embedding model, and the reason this bug survives your eval set is worse than the bug.

A bad split silently inflates every metric you'll ever report, and the bugs are subtle: leaked groups, shuffled time, preprocessing fit on everything. Here is the split done right.

Asking a model which answer is better is one line of code. Getting a number you would let block a deploy takes position-bias control, a confidence interval, and a human-labeled set the judge is measured against. Here is the harness.

Every RAG system starts here, and most candidates ship the same bug: they size the chunk in characters and call it tokens. Below is running code for all four strategies, plus the infinite loop in the overlap arithmetic and the one in the recursion.

Writing the regexes takes ten minutes. The half of the question that separates candidates is the evaluation: a detector that blocks 'ignore the noise in the data' has shipped a bug to every analyst using your product. Here is the layered detector and the harness that proves it works.

Anyone can call a rerank endpoint. The word doing the work in this question is 'prove': you have to implement DCG, ideal DCG, and nDCG@k from graded labels and report the before-and-after. Plus the ceiling nobody mentions until it bites them in production.

Asking a model 'is this answer hallucinated?' barely beats a coin flip. This is the build: claim extraction, a judge forbidden from using world knowledge, a denominator that excludes hedges, and one model call per claim, which decides where the check can run at all.

A 22-point AUC jump is both an opportunity and a red flag. Weaker candidates cheer; sharp ones grow wary and know precisely which leakage checks belong before anything ships.

The interviewer wants to see whether you choose metrics to fit the problem or just recite definitions. The signal that counts is recognizing when accuracy and even AUC mislead, and linking each metric to a decision.

Imbalance turns up in fraud, churn, and abuse, and the naive answer (oversample, done) leaks data and inflates offline metrics. The signal is favoring cost-sensitive learning, fixing the metric, and resampling correctly. Here is the full toolkit with its tradeoffs.

A point-estimate win on a metric is not a genuine win. The signal is testing significance with the right paired method on a clean held-out set, weighing effect size, and confirming online. Here is how to compare models without fooling yourself.

Ranking is not classification, so accuracy is the wrong instrument. What you need to show is matching the rank-aware metric to the task: one right answer versus many, binary versus graded relevance. Here is the answer.

Regression calls for different metrics than classification, and each tells its own story. What matters is knowing what RMSE vs MAE vs R² vs MAPE really measure and their failure modes around outliers, scale, and zeros.

A 92% aggregate accuracy can mask 60% on the segment that matters most. Averages are exactly where these failures stay buried. Here is how to surface them and the menu of fixes that actually map to the cause.

When labels exist only for the cases you already acted on, the model learns a distorted world: strong offline, blind to everyone you never saw. Worse, its own decisions choose the next labels. Here is how to spot it and counter it.

The built-in importance scores from XGBoost can rank a random ID above a vital feature, and stakeholders will base decisions on that bar chart. Here is why default importance misleads and what a careful answer reports instead.

Two models landing at the same accuracy is common, and accuracy was rarely the deciding factor to begin with. The real tiebreakers are the ones a junior candidate leaves out. This is the checklist a staff engineer works through before shipping either one.

'The model got worse' can mean three very different things with three different fixes. Retrain a model that a pipeline bug actually broke and you just bake in garbage. Here is the order to triage it in.

Most real datasets carry wrong labels, and they quietly cap your accuracy. Hand-cleaning all of it does not scale. Here is how to surface the bad labels and train robustly around them.

In search and recommendation, A/B tests can be slow and noisy because they pit different users against each other. Interleaving compares two rankers inside the same user's results and spots winners with far less traffic. Here is how.

Most teams run human eval as a vibe check with ten examples and a 1-5 slider. The signal is treating it as a designed experiment: pairwise, blinded, powered, and agreement-measured. Here is the protocol that survives scrutiny.

Accuracy against a held-out label is the wrong instrument for a system with no single right answer, an open failure space, and a metric that is itself a model. Here is what breaks, what survives, and what replaces it.

A model that never looks at the image can score respectably on VQA, which tells you most of what you need to know about the benchmark. Here is what the scores hide, and the one-line ablation that exposes it.

An eval suite that never varies the image will happily pass a model that has stopped looking at it. Start with the modality ablation, then pick metrics by what the system actually produces. Here is the full stack.

The most common ML system design round. Interviewers reward the funnel structure: candidate generation, then ranking, then re-ranking, with the right model at each stage plus an honest plan for cold start, freshness, and feedback loops. This lays out that structure.

A Devin-style agent that carries an issue all the way to a merged PR is the hardest kind to make reliable, since every step can fail and errors pile up. Here is the architecture and the loop that keeps it honest.

When every team evaluates prompts in one-off notebooks, an org ships regressions and argues about vibes. A shared eval platform turns quality into something measurable and comparable. Here is what it must provide.

An AI support agent working through millions of chats will occasionally be wrong, rude, or unhelpful, and no one can read them all. Watching quality at that scale is a design problem in itself, with its own failure modes.

This asks you to design the scoreboard, not the player. A strong answer keeps offline gates (AUC, calibration, NDCG) apart from the online verdict (A/B with revenue, user, and advertiser guardrails), and layers in counterfactual replay so you can trust a model before it ever serves a live auction.

Summarizing a 200-page contract is not one LLM call: it is chunking, hierarchical reduction, and a faithfulness check so you never invent facts. Learn the map-reduce pattern, when long-context wins out, and how to evaluate summaries at scale.

A writing assistant has to feel instant as the cursor moves through a live document. Learn how to separate fast deterministic checks from LLM rewrites, stream suggestions, and keep edits stable so the text does not flicker while the user types.

Auto-resolving support tickets means grounding answers in your knowledge base, safely calling real APIs (refunds, account changes), and knowing when to hand off to a human. Learn the agent loop, the guardrails, and how to measure resolution without eroding trust.

Selling RAG as a product means wrangling messy ingestion, multi-tenant isolation, and per-customer index freshness, all behind a simple API. See the ingestion and query planes, how to keep tenants isolated, and how to hand customers eval and observability.

Moving a cloud-API POC into a bank or defense network kills your model provider, your judge API, and your vendor dashboards in one stroke. Learn the signed-bundle supply chain, local evaluation loop, and zero-egress observability that make an air-gapped LLM stack actually operable.

The LLM is the easy part. The system is a learner model: a skill graph, a per-student mastery estimate, and an item selector that keeps the student at the edge of their ability. Here is the design, including the pedagogical constraint most candidates miss.

Five thousand businesses, five thousand assistants, one platform. The hard parts are tenant isolation you can defend, noisy neighbors, onboarding a tenant with zero data, and evaluating assistants you can never manually QA. Here is the control plane and the data plane.

Trust is an interface and reliability problem, not a messaging one, and the goal is not maximum trust. Here is the mechanism list that actually moves it, the recovery path after a public failure, and the metric most teams optimize in the wrong direction.

You cannot improve an LLM app you cannot see into, and LLM observability is not service monitoring. What counts is tracing multi-step chains, capturing inputs/outputs/tokens/cost, and online quality signals, not just latency and errors.

LLMOps is not MLOps with bigger models. What counts is the genuinely new surfaces (prompts as deployable artifacts, eval-driven development, often no training step) and treating prompts with the same version discipline as code.

Editing a prompt is a code change that ships without a compiler or a unit test by default, so quality regressions slip out invisibly. Treating prompts and models as testable artifacts is what divides a toy from a product. Here is the harness.

Migrating models is a project you plan. Being migrated is a Tuesday. When the weights move under a stable API and your prompts were tuned against the old ones, every dashboard stays green while quality quietly rots.

FP8 serving is fast and usually fine, until it quietly degrades on the one workload your benchmark did not cover. What counts is knowing what FP8 breaks and how to validate it. Here is the recipe.

There is no one-shot patch for prompt injection, and answering 'sanitize the input' loses the round. What lands is defense in depth: privilege boundaries, handling retrieved content as data rather than instructions, and a human gate on irreversible actions. Here is the layered answer.

Fairness questions catch engineers who treat it as a vibe. What lands is knowing the formal fairness metrics conflict mathematically, that you have to choose one deliberately for the context, and where bias enters the pipeline. Here is the rigorous, honest answer.

A classic ML-security question that trips up people who only know clean-data accuracy. What lands is grasping why models are brittle, the realistic threat model, and the fact that no single defense holds. Here is the honest answer.

Red teaming is how you uncover an AI system's failures before users or attackers do. What lands is a structured program covering the real attack surface (jailbreaks, harmful content, leakage, bias) and routing every finding back into defenses.

The number is bait. Interviewers use this to see whether you treat a benchmark delta as a decision or as a claim to be audited, and whether you can price latency, cost, and on-call burden against accuracy. Here is the answer that scores.

The classic closer, and most candidates answer it with a horoscope or a list of tool names. The signal is whether you hold an actual view, can justify it from what is already happening, and know what it means for the job.

Most candidates answer this with a speech about tradeoffs. The ones who get hired describe a mechanism: the teams that ship fastest in AI are not the ones with fewer guardrails, they are the ones whose guardrails make a mistake cheap.

The ordering is the answer, and one move separates the people who have shipped from the people who have prototyped: where evaluation sits in the sequence. Most candidates put it in the wrong place and lose the question in the first sentence.

Training a pCTR model is the easy part. The challenge is proving a change helps before it reaches revenue, when the model runs inside an auction, the logs surface only ads that won, and a 1% calibration error costs real money. This is an eval question, not a model question.