TL;DR: Tier it: cheap exact-match filters (perceptual hashes of known-bad content, rules) handle the obvious, ML classifiers score the rest, and uncertain or high-severity cases go to human review. Set the action by severity and confidence: auto-remove only at high confidence on severe content, otherwise downrank, age-gate, or queue for humans. Build for an adaptive adversary, multilingual and multimodal content, appeals, and reviewer well-being. No single threshold works; you balance precision against recall per policy and harm level.
How to approach it
Pin down the policy categories (spam, hate, violence, CSAM, self-harm) and establish that the right action varies by category and severity, and that latency matters: do you moderate before content becomes visible or right after. Then walk through the tiered pipeline and focus your time on the per-severity precision/recall calibration and human-in-the-loop, where moderation is genuinely hard. Do not lead with a model choice; lead with the asymmetric cost of errors.
A strong answer
The pipeline is tiered so cost rises only as ambiguity rises.
- Tier 1, exact match. Known-bad content (especially CSAM and terrorist material) is matched by perceptual hashes against shared databases like PhotoDNA, in microseconds and with near-zero false positives. It is the cheapest and most certain layer, so it runs first.
- Tier 2, ML classifiers. Per-policy models score the long tail: text toxicity and hate classifiers, image nudity and violence detectors, increasingly multimodal models. These are probabilistic, so they feed thresholds, not hard decisions.
- Severity-and-confidence routing (the crux). The cost of errors is asymmetric and category-dependent. A false negative on CSAM or imminent self-harm is catastrophic, so you favor recall and escalate aggressively. A false positive on borderline speech wrongly silences a user, so you favor precision and prefer downranking over removal. That is why one threshold is wrong: you auto-act only where confidence and severity both justify it, and otherwise apply softer actions (downrank, limit reach, age-gate, warn) or route to humans.
- Human-in-the-loop. Humans handle ambiguous and high-severity cases, produce the labels that retrain the models, and serve as the appeals backstop. Design the queue for throughput and for reviewer well-being, with exposure limits and grayscaling on graphic content.
- Adversarial and multilingual reality. Bad actors adapt: leetspeak, coded language, adversarial perturbations on images. Content spans languages and modalities. So you monitor drift, continuously inject adversarial examples into training, and cover languages explicitly rather than assuming an English model generalizes.
Measure precision and recall per policy at the live operating point, never aggregate accuracy. Track measured harm (prevalence of violating content that slipped through), reviewer load, appeal overturn rate, and latency.
Key takeaways
- Tier by cost and certainty: hashes first (exact, near-zero FP), ML for novel content, humans for ambiguity and severity.
- The decision is a two-axis function of severity and confidence, not a single global threshold.
- Report precision/recall per policy at the operating point plus measured harm prevalence, not accuracy.
- Treat it as an arms race: drift monitoring, continuous adversarial data, and a real appeals loop that feeds labels back.
What interviewers probe next
- "Precision or recall, and where?" It depends on harm: recall-favoring with escalation for severe categories (missing it is catastrophic), precision-favoring softer actions for speech (false positives censor). One global threshold is wrong.
- "Why hash matching if you have ML?" For known content it is instant, exact, and shareable across platforms; ML covers novel content the hashes have never seen.
- "How do you handle appeals and mistakes?" A real appeals path with human review, and overturns feed back as labels. This is both fairness and a data source.
- "Adversaries evolve, how do you keep up?" Continuous red-teaming and adversarial data, drift monitoring, and fast model refresh; treat it as an arms race, not a one-time training run.
Common mistakes
- One model and one threshold for all policies, ignoring that error costs differ wildly by category.
- Auto-removing on low-confidence ML scores, over-censoring legitimate content.
- No human-in-the-loop for ambiguity, high severity, or appeals.
- Assuming an English or text-only model covers a multilingual, multimodal platform, and ignoring adversarial adaptation.