deduplication

Fine-tuning lives or dies on data, not on hyperparameters. What shows depth is spelling out what makes a set good (quality, diversity, format, dedup) and defending why a few thousand clean examples outperform a million noisy ones.

Top-k by raw similarity often feeds the model five paraphrases of one paragraph, denying it the other facts the question needs. The fix is ranking for relevance and novelty together. Here is how.

Comparing all pairs across a million documents is a trillion comparisons. MinHash estimates Jaccard similarity from a tiny signature, and LSH converts dedup into a near-linear scan. Here is the implementation.

A Bloom filter answers 'have I seen this?' with a few bits per item rather than storing the key, swapping a rare false positive for a large memory saving. Here is the build, the sizing math, and where it pays off in dedup and serving.

A grounded systems-coding screen: locate files with identical content across a tree. The naive all-pairs hash wastes work. The signal is the size prefilter plus a cheap-hash gate ahead of the full hash. Here is the layered approach.

A hard streaming-systems question: dedup at high throughput while keeping state bounded. What interviewers reward is a tiered state design (a probabilistic filter ahead of durable state) plus watermark-driven eviction. Here is the architecture that does not OOM.

Dedup is a daily data-engineering task and a frequent screen. The signal is ROW_NUMBER to keep the right record (not just DISTINCT), pinning down what 'duplicate' actually means, and handling near-duplicates at scale.

Anyone can say 'filter and dedup.' The signal is the funnel arranged by cost, the MinHash/LSH banding math, and recognizing that the shuffle across billions of documents is what actually runs up the bill, plus the benchmark decontamination people forget until their eval numbers get challenged.

Crawling a few pages is trivial. Crawling the web means a URL frontier with priority, per-host politeness, dedup across billions of URLs, and defenses against spider traps and infinite content. Here is the crawler architecture that scales without getting your IPs banned.

Large models memorize and can be coaxed into emitting training examples verbatim, a genuine privacy and copyright liability. The defenses reach across data, training, and output. Here is the layered answer.