counterfactual

You observe repayment only for applicants you approved, so your training data is censored by your own past policy and your offline test set is censored the same way. The fix starts with something you must have logged years ago.

This asks you to design the scoreboard, not the player. A strong answer keeps offline gates (AUC, calibration, NDCG) apart from the online verdict (A/B with revenue, user, and advertiser guardrails), and layers in counterfactual replay so you can trust a model before it ever serves a live auction.

Training a pCTR model is the easy part. The challenge is proving a change helps before it reaches revenue, when the model runs inside an auction, the logs surface only ads that won, and a 1% calibration error costs real money. This is an eval question, not a model question.