watermarks

Pipelines fail and retry, so the real question is whether a retry corrupts your data. What interviewers look for is idempotent writes (not 'exactly-once delivery') alongside watermarks for late data. Here is how to keep re-runs safe without double-counting.

A hard streaming-systems question: dedup at high throughput while keeping state bounded. What interviewers reward is a tiered state design (a probabilistic filter ahead of durable state) plus watermark-driven eviction. Here is the architecture that does not OOM.

You cannot wait for all the data, and buffering an unbounded stream will OOM. The signal is windowed joins with watermarks for stream-stream, lookup joins for stream-table, and a clean account of late and out-of-order events.

Events arrive minutes or days after they happened, and a window that has already closed reports wrong counts. The signal is event-time vs processing-time, watermarks with allowed lateness, and how you fix already-emitted aggregates.