calibration

Every customer wants to know how to stop the model inventing things. The weak reply is 'better prompts.' The signal is understanding why models hallucinate and stacking grounding, abstention, and measurement into a system you can defend.

LLM-as-judge is how you grade open-ended generation at scale, and G-Eval put a formal frame on it. What shows depth is naming the biases that make raw judge scores untrustworthy and demonstrating exactly how you harden them.

Models are trained to be helpful, which quietly teaches them to never refuse. Restoring honest abstention is a system problem, not a prompt tweak. Here is the stack that actually shifts the refuse-when-unsure rate.

Push PPO hard enough and true quality peaks then declines while the reward keeps climbing. The signal is knowing the Gold-vs-proxy gap, the KL budget that bounds it, and how you actually measure when to stop.

Pairwise asks which one wins, pointwise asks how good a single output is. The signal is knowing which is more reliable, how Elo/Bradley-Terry aggregates comparisons, and where each method quietly misleads.

A reward model converts pairwise preferences into a scalar signal RLHF can optimize. The signal is the Bradley-Terry loss, the base-model and head choices, and how you validate it before trusting it.

Definitions take thirty seconds. The interview is about the four things log-probs buy you in production: a confidence signal, banning tokens, guaranteeing valid output, and classification you can calibrate. Plus the tokenization gotcha that quietly breaks all of them.

A one-word rewrite moving your accuracy several points is not a prompting problem, it is an uncalibrated decision function. The fix is measuring the model's prior over labels and dividing it out, not another round of tinkering.

A model can rank perfectly and still output meaningless probabilities. The signal is knowing precisely when calibration matters, how to measure it, and the post-hoc fixes. Here is the answer most candidates miss.

Two inexpensive regularizers that cure overconfident classifiers. What proves you understand them is knowing that one softens the target while the other softens the input, and precisely why gentler signals produce calibrated models. Here is the answer.

Wrap any model and obtain prediction sets with a provable coverage rate, with no distributional assumptions needed. What shows depth is the calibration-set plus nonconformity-score mechanism and exactly what the guarantee does and does not promise. Here is the answer.

A model can rank perfectly yet lie about its probabilities; its 0.9 may hold only 70% of the time. What shows depth is knowing the three fixes, the data size each requires, and how to measure the gap. Here is the answer.

Defaulting to 0.5 leaves money or safety on the table. The right cutoff comes from the cost of each error and the operating constraint, not the model. Here is how to set it deliberately.

Two models landing at the same accuracy is common, and accuracy was rarely the deciding factor to begin with. The real tiebreakers are the ones a junior candidate leaves out. This is the checklist a staff engineer works through before shipping either one.

Fairness has no single number, and a well-known result proves the main three cannot hold together. What interviewers reward is defining each metric precisely and explaining the impossibility rather than picking one blindly. Here is the answer.

Ads ranking is where calibrated probabilities collide with tight latency and money. The signal is knowing that CTR has to be calibrated (not just ranked), the feature and serving design, and the auction context. Here is the design that goes past 'train a classifier.'

Reps only have so many hours, so the model exists to steer them toward the leads worth calling. Interviewers listen for value-weighted prioritization, carefully built conversion labels, and evidence of revenue uplift measured against a holdout.

This asks you to design the scoreboard, not the player. A strong answer keeps offline gates (AUC, calibration, NDCG) apart from the online verdict (A/B with revenue, user, and advertiser guardrails), and layers in counterfactual replay so you can trust a model before it ever serves a live auction.

Trust is an interface and reliability problem, not a messaging one, and the goal is not maximum trust. Here is the mechanism list that actually moves it, the recovery path after a public failure, and the metric most teams optimize in the wrong direction.

You cannot wait for ground truth to tell you the model broke. Prediction-side monitoring catches failures in minutes, not weeks, provided you know which output signals actually move first.

Quantization is the opening move for shrinking and accelerating models, and the interviewer expects more than 'use fewer bits.' What they grade is whether you know what each precision level gains you, why outliers wreck naive quantization, and how you demonstrate quality survived. This lays that out.

Naive 4-bit quantization can wreck quality, and the reflex to give up and serve fp16 leaves a large speedup unclaimed. The accuracy is usually recoverable. Here is the ladder of fixes.

FP8 serving is fast and usually fine, until it quietly degrades on the one workload your benchmark did not cover. What counts is knowing what FP8 breaks and how to validate it. Here is the recipe.

Training a pCTR model is the easy part. The challenge is proving a change helps before it reaches revenue, when the model runs inside an auction, the logs surface only ads that won, and a 1% calibration error costs real money. This is an eval question, not a model question.

Watch time is the label everyone optimizes and hardly anyone measures cleanly. You see minutes only for videos people chose to play, the distribution is savagely skewed, and the slot they saw it in shifted the number. The interview asks whether you can predict a biased label honestly.

This model's output is a price input, not a ranking. A miscalibrated CTR skews the auction bid, so you overpay or underdeliver. Throw in conversions that arrive days after the click and advertisers with no history, and calibration plus delayed feedback become the entire interview.