Idempotency and Exactly-Once Effects

TL;DR: Networks fail and clients retry, so the same request can land more than once. Idempotency means running a request twice has the same effect as once, so retries are safe. You get it with idempotency keys (the client sends a unique ID; the server records processed IDs and ignores duplicates) and deduplication. Without it, a retried payment charges twice, a retried pipeline double-counts. True "exactly-once" is usually at-least-once delivery plus idempotent processing, you cannot guarantee a message arrives exactly once, so you make duplicates harmless.

Why duplicates are inevitable

In any distributed system, a request can succeed on the server while the response is lost in transit, so the client (unaware the write landed) retries. The operation now runs twice. Retries with backoff are essential for resilience, yet they guarantee duplicates. The real question is not "how do I avoid retries" but "how do I make a repeated request safe."

This bites hardest at boundaries you do not control. A payment API times out at 30s, your gateway already retried, the bank actually processed both attempts. A Kafka consumer crashes after writing to the DB but before committing its offset, so on restart it reprocesses the same batch. An LLM tool-call to send_email returns a 504 even though the provider sent the mail. In every case the effect happened, the acknowledgment did not.

Idempotency keys and dedup

The standard mechanism:

• The client attaches a unique idempotency key to the request (a UUID minted once per logical operation, reused across retries of that same operation).
• The server checks whether it has already processed that key; if so, it returns the stored result without re-executing; if not, it processes once and records the key.
• So even if the request arrives five times, the effect happens once.

The subtlety most candidates miss: the check-and-record must be atomic, or two concurrent retries both see "not processed" and both execute. In practice you do a conditional insert on a unique key column (INSERT ... ON CONFLICT DO NOTHING, or SETNX in Redis) so the database, not your code, serializes the race. Stripe's idempotency layer works exactly this way: keys are scoped per account and retained ~24 hours, and a replay returns the original response byte-for-byte.

Reads are naturally idempotent; the work is making writes/effects (payments, sends, inserts) idempotent. Techniques: unique keys, upserts keyed on a business id (see idempotent pipelines), and conditional writes (UPDATE ... WHERE version = n).

Worked example. A checkout service charges $40 and emits a fulfillment event. Client times out, retries 3 times.

The lesson: idempotency is not one switch. Every side effect on the path needs its own dedup, or the weakest link multiplies.

"Exactly-once" really means

A common misconception is that systems deliver messages "exactly once." In practice you get at-least-once delivery (messages may repeat) or at-most-once (messages may be lost). The reliable pattern is at-least-once delivery plus idempotent processing: accept that duplicates arrive, and design processing so duplicates have no extra effect. That combination behaves like exactly-once effects, which is what you actually care about. Even Kafka's "exactly-once semantics" is at-least-once delivery plus transactional dedup under the hood, not magic on the wire.

Why interviewers probe this

LLM and data pipelines are full of flaky, retried steps (provider timeouts, restarted jobs), so idempotency is a constant real concern, and "a retry double-charged the customer" is a classic incident. A strong answer explains why duplicates are inevitable (lost responses, retries), how idempotency keys/dedup make repeats safe, the atomic check-and-record detail, and that "exactly-once" is at-least-once-delivery plus idempotent-processing. That shows you design for failure, the core of reliable systems.

Common misconceptions

• "Exactly-once delivery is achievable." Delivery is at-least-once (or at-most-once); you get exactly-once effects via idempotent processing.
• "Retries are the problem." Retries are necessary; the fix is making processing idempotent, not avoiding retries.
• "One idempotency key fixes the request." Each side effect on the path needs dedup, or the unguarded one multiplies under retry.
• "Check-then-write is enough." Without an atomic conditional insert, two concurrent retries both pass the check and both execute.
• "Idempotency is only for payments." Any retried side effect (pipeline writes, notifications, model tool-calls) needs it.

Key takeaways

• Retries make duplicate requests inevitable; idempotency makes processing twice equal processing once.
• Achieve it with idempotency keys plus deduplication, or upserts keyed on a business id, with an atomic check-and-record.
• Guard every side effect on the path, not just the first one.
• "Exactly-once" in practice is at-least-once delivery plus idempotent processing (exactly-once effects).