ab testing

Offline metrics gate releases; online metrics reveal what actually happened with users. What shows depth is understanding why offline wins routinely fail to hold up online, and how the two feed each other.

Shipping a model is itself an experiment, and this question tells apart people who run A/B tests from those who p-hack them. The signal is pre-registering the metric, sizing the test, and holding back from peeking.

Hypothesis testing sits under every experiment, and what matters is reading the p-value correctly (it trips most people) and picking the right test. Here is the rigorous, plain-language answer.

A bandit learns and optimizes at the same time, moving traffic toward the winning arm during the experiment rather than holding the split fixed like an A/B test. Interviewers look for the explore-exploit tradeoff, the three core algorithms, and a clear sense of when a bandit wins over a clean A/B test.

Running an A/B test is easy; running a valid one is hard. What matters is naming the traps that produce confidently wrong conclusions and the fix for each. Here is the checklist a senior experimenter carries.

A pre-existing gap between your groups means randomization or instrumentation is broken, and the entire experiment is suspect. Strong candidates spot sample-ratio mismatch immediately. Here is the full diagnosis and the disciplined response.

In search and recommendation, A/B tests can be slow and noisy because they pit different users against each other. Interleaving compares two rankers inside the same user's results and spots winners with far less traffic. Here is how.

This asks you to design the scoreboard, not the player. A strong answer keeps offline gates (AUC, calibration, NDCG) apart from the online verdict (A/B with revenue, user, and advertiser guardrails), and layers in counterfactual replay so you can trust a model before it ever serves a live auction.

Running experiments on LLM features is tough because outputs are open-ended and quality is fuzzy. See how to assign traffic, choose metrics beyond engagement, tame variance from non-determinism, and dodge the traps that let a winning variant lose once it ships.

A dependable experiment platform goes well beyond splitting traffic in half. What matters is stable assignment, exposure logging, statistical discipline, and guardrails that hold up against peeking and sample-ratio mismatch. Here is the design.

Shadow deployment validates a new model on real traffic without exposing users to it. What matters is knowing outputs run in parallel then get discarded, and how it complements canary and A/B. Here is the answer.

Champion-challenger is how you improve a production model without betting the business on a hunch: the live model versus candidates competing on real metrics. What matters is the ongoing-competition framing and disciplined, pre-committed promotion.

Training a pCTR model is the easy part. The challenge is proving a change helps before it reaches revenue, when the model runs inside an auction, the logs surface only ads that won, and a 1% calibration error costs real money. This is an eval question, not a model question.

New courses launch every week and most browsers are signed-out or brand new, so the interesting half of this problem is what you recommend when you have almost no behavioral signal. The other half is avoiding the trap of optimizing engagement into a clickbait catalog.

This product succeeds or fails on a feedback loop measured in seconds: every swipe is a label, and the following video must respond to it. The tricky part is exploring widely enough to learn what someone enjoys while avoiding a filter bubble or the amplification of junk.

A search box is a recommender carrying a strong prior: the query. The difficult pieces are reading ambiguous intent, retrieving in two stages within a tight latency budget, and learning from clicks without teaching the model that whatever appeared first is best.