Category: distributed systems Difficulty: 4 Asked: 2026-10-31 · distributed-systems-01 · Distributed systems · Edit
Tip: answer aloud for ~5 minutes before reading the solution.
Question
A mobile client calls POST /charges; the network times out and the client retries. How do you guarantee the customer is charged exactly once, end to end, including the call to an external card processor?
Step-by-step solution
State the truth: networks give at-most-once or at-least-once; "exactly once" = at-least-once delivery + idempotent processing.
Client generates an idempotency key (UUID) per logical operation and resends it on every retry, with exponential backoff + jitter.
Server stores key → (request fingerprint, status, response) in the primary DB with a unique constraint, written in the same transaction as the business effect.
Retry with the same key: return the stored response; same key with a different payload → 4xx error; concurrent in-flight duplicate → 409/lock.
Break work into atomic phases with recovery points (Brandur's pattern): local DB work in transactions; the foreign call (processor) is the non-transactional step — pass your own idempotency key downstream so the processor dedupes too.
A background completer/reaper resumes or fails stuck requests; keys expire after e.g. 24h.
Final answer
Client-supplied idempotency key, persisted atomically with the result under a unique constraint; replays return the saved response; multi-step work is split into recoverable atomic phases, and the key is propagated to the processor so every hop is idempotent.
Trade-offs & common mistakes
Storing the key in a cache separate from the DB write → the two can diverge.
Generating the key server-side (retries get new keys).
Not fingerprinting the payload (key reuse with different amounts).
Retrying without jitter → thundering herd.
Claiming exactly-once delivery exists in the network.
Key terms
idempotency key, at-least-once delivery, atomic phase, backoff with jitter