Your churn model's AUC jumps from 0.71 to 0.93 after adding a 7-day rolling feature. What now?

Question

AI Interview Training · Accepted Answer

TL;DR: Treat a 22-point AUC jump from a rolling feature as leakage until you prove otherwise. Temporal leakage is the usual cause: the rolling window pulls in data from at or after prediction time. Apply an as-of cutoff so each feature draws only on data strictly before the label time, move to a time-based split, and check that the lift holds. Genuine gains survive a chronological split; leakage does not. How to approach it. Hold off on the celebration. State your suspicion (temporal leakage) and the mechanism behind it (the rolling feature peeking past prediction time), then walk through the checks in sequence: feature construction, split strategy, and a clean holdout that mimics deployment. Treat all of it as a single question: what would make this number honest. A strong answer. A jump that large almost never reflects a genuinely better feature; it usually means the feature encodes the label. With a 7-day rolling aggregate the classic bug is computing the window relative to "now" (training time) instead of each row's prediction time. For a user who churned on day 5, the window then quietly includes post-churn behavior, or includes no behavior at all, and that absence is itself the signal. The fix is an as-of join: every feature for a prediction at time t may use only events with timestamp strictly less than t. Then change evaluation. K-fold with random shuffling leaks future into past, which is exactly how a leaky feature looks great offline. Use a time-based split: train on [0, tcut), validate on [tcut, tcut + h), and keep a final holdout that mimics the real prediction cadence. Re-run the model with and without the rolling feature under that chronological split. If AUC collapses back toward 0.71, the lift was leakage. If a plausible gain survives (not 0.93), the feature is real and you keep it. Run these leakage checks alongside the split fix: inspect feature importance (one feature that dominates everything is a tell), compare the feature's distribution for churned vs retained users near the label boundary, and confirm the feature is even computable at serving time from past data only. A 0.93 AUC on churn is suspicious on its face; honest churn models usually land far lower. Key takeaways A 20+ point jump from a rolling feature is a leakage hypothesis, not a win, until a chronological split confirms it. The root cause is windows anchored to training time instead of each row's prediction time; an as-of join with ts < t fixes it. Random k-fold is the wrong evaluator for temporal data; it hides exactly the leakage you are hunting. Sanity-check the absolute number: 0.93 churn AUC is implausibly high and should trigger scrutiny by itself. What interviewers probe next. "Other leakage sources besides time?" Target leakage (a feature derived from the label), train/test contamination (the same user in both splits), and preprocessing fit on the full dataset (a scaler or target encoder fit before the split). "How do you prevent this systematically?" A point-in-time-correct feature store with as-of joins, and time-aware CV as the default for any temporal problem. "AUC limitations here?" Under heavy imbalance, AUC-ROC can look strong while the model is useless at the operating threshold; report PR-AUC and calibration too. Common mistakes. Treating the jump as success and shipping it. Defending it with random k-fold numbers, the very split that hides temporal leakage. Fixing the split but leaving the feature computed relative to training time rather than each row's prediction time. Ignoring that a 0.93 churn AUC is implausible on its face.