idempotency

An hour-long agent run is a distributed workflow wearing an LLM hat. The signal is designing serializable state with step-level checkpoints, and knowing why resume is easy while replaying a side effect is not.

Every agent is a loop around this function. The candidates who fail it raise an exception on a bad tool call; the ones who pass hand the error back to the model as a tool result and let it fix itself on the next turn.

SCD2 tells apart engineers who have actually run pipelines from those who have only read about them. The answer rests on a single atomic MERGE that closes the prior row while opening the new one, and holds up when the job retries.

Pipelines fail and retry, so the real question is whether a retry corrupts your data. What interviewers look for is idempotent writes (not 'exactly-once delivery') alongside watermarks for late data. Here is how to keep re-runs safe without double-counting.

A backfill that double-counts or knocks over the live pipeline is worse than the bug it fixes. The signal is idempotent partition-scoped rewrites, keeping backfill compute isolated from live runs, and validating before the swap.

A celebrity posts and 50M followers need a notification. The tricky parts are not sending one message; they are fan-out strategy, deduplication, channel routing, and not flooding users. Here is the architecture that handles both the long tail and the viral spike.

In payments, a duplicate charge or a lost credit is not a bug, it is a financial incident. The design turns on idempotency keys, double-entry bookkeeping, and an append-only ledger you can audit and reconcile. Here is how money systems stay correct.

Background jobs fail, retry, and get redelivered. If a job sends an email or charges a card, running it twice is a real problem. The design combines at-least-once delivery with idempotent handlers, visibility timeouts, and dead-letter queues. Here is how to make jobs safe.

Anyone can wire tasks into a DAG. What matters is the patterns that keep it correct under retries and backfills: idempotency, data-aware triggering, and the asset model. Here is what separates a flaky pipeline from a trustworthy one.