TL;DR: Two layers: fast deterministic rules for known patterns and an ML model for the long tail, both scored inline in tens of ms. Skip accuracy and even ROC-AUC as targets. Set the operating threshold from the dollar cost of false positives (blocked good customers) versus false negatives (fraud loss), report PR-AUC, and address imbalance with class weights rather than naive resampling. Build for delayed, biased labels and an adversary who keeps shifting.
How to approach it. Nail down three things at the start: the latency budget (scoring runs inline with the authorization, often under 50 ms), the cost asymmetry (a blocked legitimate customer may churn, a missed fraud becomes a chargeback), and the timing of labels (chargebacks arrive days to weeks later). Then sketch the architecture and devote most of your time to imbalance and thresholding, where this question is won or lost.
A strong answer. Architecture.
transaction --> feature lookup (online store: velocity, device, history)
--> rules engine (hard blocks/allows for known patterns) [~ms]
--> ML risk model (gradient-boosted trees) -> score [~ms]
--> decision: allow / step-up auth / block, from threshold
--> async: human review queue + label capture (chargebacks)
A rules layer catches known fraud cheaply and gives explainable hard blocks (a compliance and appeals requirement, not just speed). The ML model handles the long tail. Features come from a low-latency online store: velocity counts (transactions per card per hour), device and IP signals, account age, amount versus the user's own history. Score inline within the budget.
Imbalance, the crux. With under 1% positives, accuracy is meaningless: a model that predicts "never fraud" scores 99%. ROC-AUC flatters too, because the huge true-negative pool drowns out false positives. Report PR-AUC and precision/recall at the chosen threshold. Handle imbalance with class weights or cost-sensitive loss (preferred: no data thrown away) rather than SMOTE, which fabricates synthetic points in a space where fraud is non-stationary. If you resample, do it inside the cross-validation fold, never before the split.
The decision step is the real test. Do not block at 0.5. Derive the operating point from a cost matrix: expected loss of a missed fraud against expected cost (and downstream churn) of blocking a good customer. Usually you do not binary-block at all. Route medium-risk to a step-up challenge (3-D Secure, 2FA), reserve hard blocks for high confidence, and auto-approve the rest.
| Decision | Trigger | Cost if wrong |
|---|---|---|
| Auto-approve | score below low threshold | missed fraud (chargeback) |
| Step-up auth | mid band | friction, some abandonment |
| Hard block | score above high threshold | blocked good customer, churn |
Feedback and adversary. Labels are delayed (chargebacks) and biased: you never learn whether a blocked transaction was truly fraud, so the training set is selection-skewed. Build a review queue for human labels and hold out a small unblocked control to estimate true rates honestly. Fraud is adversarial, so patterns shift weekly. Monitor score-distribution drift and retrain on a fast cadence with drift triggers.
Monitoring. Track precision and recall at threshold, dollar loss, false-positive (good-customer) rate, latency p99, and score-distribution drift.
Key takeaways
- Report PR-AUC and precision at the operating threshold; accuracy and ROC-AUC both lie under 1% positives.
- Set the threshold from a dollar cost matrix, not 0.5, and use a step-up tier for the gray zone.
- Prefer class weights or cost-sensitive loss over SMOTE; if you resample, keep it inside the CV fold.
- Labels are delayed and biased: keep an unblocked control and a review queue to measure truth.
What interviewers probe next.
- "Precision or recall more important?" Depends on costs, but you usually cannot block aggressively without harming good customers, so tune to a precision target and route the gray zone to step-up auth.
- "Why GBMs over deep nets here?" Tabular features, strong performance, fast inference, and interpretability for review and appeals. Deep nets earn their keep on sequence or graph signals.
- "Delayed labels: how do you evaluate now?" Proxy metrics (review-queue hit rate, score drift) plus a labeled holdout. Accept that ground truth lags.
- "Graph or ring fraud?" Add graph features (shared devices and cards across accounts) to catch coordinated rings that point rules miss.
Common mistakes.
- Reporting accuracy or ROC-AUC and ignoring precision at the operating threshold.
- Defaulting to a 0.5 cutoff instead of deriving the threshold from costs.
- SMOTE or oversampling applied before the train/test split, leaking and inflating offline metrics.
- Ignoring label delay and adversarial drift, so the model silently decays.