minhash

Comparing all pairs across a million documents is a trillion comparisons. MinHash estimates Jaccard similarity from a tiny signature, and LSH converts dedup into a near-linear scan. Here is the implementation.

Anyone can say 'filter and dedup.' The signal is the funnel arranged by cost, the MinHash/LSH banding math, and recognizing that the shuffle across billions of documents is what actually runs up the bill, plus the benchmark decontamination people forget until their eval numbers get challenged.