bloom filter

A Bloom filter answers 'have I seen this?' with a few bits per item rather than storing the key, swapping a rare false positive for a large memory saving. Here is the build, the sizing math, and where it pays off in dedup and serving.

A hard streaming-systems question: dedup at high throughput while keeping state bounded. What interviewers reward is a tiered state design (a probabilistic filter ahead of durable state) plus watermark-driven eviction. Here is the architecture that does not OOM.

Crawling a few pages is trivial. Crawling the web means a URL frontier with priority, per-host politeness, dedup across billions of URLs, and defenses against spider traps and infinite content. Here is the crawler architecture that scales without getting your IPs banned.