Practice: answer each aloud in ~5 min before reading the solution; self-grade against "Final answer" and "Common mistakes"; add misses to REVIEW.md.
Design a distributed API rate limiter
Design a distributed API rate limiter
Category: system design Difficulty: 4 Asked: 2026-10-31 · system-design-01 · System design · Edit
Tip: answer aloud for ~5 minutes before reading the solution.
Question
Design rate limiting for a public API serving ~1M req/s across multiple regions: per-API-key limits, burst tolerance, and protection of the platform itself under overload. Where does it live, what algorithm, what store, and what happens when the limiter fails?
Step-by-step solution
Clarify: limit keys (API key, user, IP, endpoint), hard vs soft limits, accuracy needed, global vs per-region, response contract (429 + Retry-After, rate headers).
Distinguish request rate limiters (per customer) from load shedders (protect the fleet, drop low-priority traffic first) — Stripe runs both layers.
Choose algorithm: token bucket (allows bursts, Stripe's choice), or sliding-window counter (approximates a sliding log cheaply by weighting the previous window — Cloudflare's approach). Avoid fixed windows (2x burst at the boundary) and full sliding logs (memory).
Placement: at the edge/API gateway so rejected traffic is cheap; keep the decision path in-memory or one hop away.
State: centralized Redis with atomic Lua (INCR+EXPIRE / token math) sharded by key; or local counters per node with periodic async sync for scale, accepting slight over-admission.
Multi-region: per-region budgets (global limit / N or weighted) instead of synchronous cross-region coordination.
Failure mode: fail open on limiter-store errors (with tight timeouts) for customer limits; load shedder stays local.
Operate: dark-launch (log-only), dashboards on rejections per key, allowlists for internal traffic, client guidance (exponential backoff + jitter).
Final answer
Two layers at the edge: per-key token bucket (or sliding-window counter) backed by sharded Redis with atomic scripts, plus a local priority-based load shedder. Return 429 with Retry-After, fail open on limiter outages, partition global limits per region, and roll out in log-only mode first.
Trade-offs & common mistakes
Accuracy vs latency: exact global counts need coordination; approximate is almost always fine.
Fixed windows allow double bursts; sliding logs blow up memory.
Failing closed turns a Redis blip into a full outage.
Forgetting load shedding: per-customer limits don't save you when total traffic spikes.
Category: distributed systems Difficulty: 4 Asked: 2026-10-31 · distributed-systems-01 · Distributed systems · Edit
Tip: answer aloud for ~5 minutes before reading the solution.
Question
A mobile client calls POST /charges; the network times out and the client retries. How do you guarantee the customer is charged exactly once, end to end, including the call to an external card processor?
Step-by-step solution
State the truth: networks give at-most-once or at-least-once; "exactly once" = at-least-once delivery + idempotent processing.
Client generates an idempotency key (UUID) per logical operation and resends it on every retry, with exponential backoff + jitter.
Server stores key → (request fingerprint, status, response) in the primary DB with a unique constraint, written in the same transaction as the business effect.
Retry with the same key: return the stored response; same key with a different payload → 4xx error; concurrent in-flight duplicate → 409/lock.
Break work into atomic phases with recovery points (Brandur's pattern): local DB work in transactions; the foreign call (processor) is the non-transactional step — pass your own idempotency key downstream so the processor dedupes too.
A background completer/reaper resumes or fails stuck requests; keys expire after e.g. 24h.
Final answer
Client-supplied idempotency key, persisted atomically with the result under a unique constraint; replays return the saved response; multi-step work is split into recoverable atomic phases, and the key is propagated to the processor so every hop is idempotent.
Trade-offs & common mistakes
Storing the key in a cache separate from the DB write → the two can diverge.
Generating the key server-side (retries get new keys).
Not fingerprinting the payload (key reuse with different amounts).
Retrying without jitter → thundering herd.
Claiming exactly-once delivery exists in the network.
Key terms
idempotency key, at-least-once delivery, atomic phase, backoff with jitter
The dual-write problem and the transactional outbox
The dual-write problem and the transactional outbox
Category: messaging and events Difficulty: 4 Asked: 2026-10-31 · messaging-events-01 · Messaging and events · Edit
Tip: answer aloud for ~5 minutes before reading the solution.
Question
The Order service must save an order to Postgres and publish OrderCreated to Kafka. A team writes db.commit(); kafka.send(). What breaks, and how do you redesign it? How do consumers stay correct?
Step-by-step solution
Identify the dual write: crash between commit and send → lost event; send then rollback → phantom event. No 2PC across DB and Kafka in practice.
Transactional outbox: in the same local transaction, insert the order and an outbox row (aggregate id, type, payload).
Relay the outbox: either a poller (simple, adds latency/load) or CDC log-tailing (e.g. Debezium reading the WAL) to publish to Kafka.
Delivery is at-least-once → consumers must be idempotent (dedupe by event id / processed-messages table).
Ordering: partition by aggregate id so events for one order stay ordered.
For multi-service business transactions, chain this with a saga (compensating actions), not distributed transactions.
Final answer
Write the business row and the event into an outbox table in one ACID transaction, publish via CDC (Debezium) or a poller, key by aggregate id for ordering, and make consumers idempotent because delivery is at-least-once.
Trade-offs & common mistakes
Poller vs CDC: simplicity vs latency/operational complexity of connectors.
Proposing XA/2PC — fragile, poorly supported by brokers.
Forgetting consumer dedupe; assuming global ordering.
Category: reliability and observability Difficulty: 4 Asked: 2026-10-31 · reliability-observability-01 · Reliability and observability · Edit
Tip: answer aloud for ~5 minutes before reading the solution.
Question
As CTO you inherit a platform with hundreds of noisy CPU/latency alerts and constant tension between product velocity and stability. How do you define reliability targets, alert on them, and use them to govern release decisions?
Step-by-step solution
Pick user-centric SLIs per critical journey (availability = good requests / valid requests; latency = % under threshold at the load balancer or client).
Set SLOs below 100% based on user need and history (e.g. 99.9% over 28 days); 100% is the wrong target.
Replace cause-based alerts with multi-window, multi-burn-rate alerts (e.g. page at 14.4x burn over 1h confirmed by 5m; ticket at lower burn rates over longer windows).
Keep cause metrics as dashboards for debugging, not pages.
Review SLOs quarterly; report budget consumption to execs.
Final answer
Define SLIs on user journeys, set realistic SLOs, derive error budgets and an executive-backed policy that ties budget burn to release velocity, and page only on multi-window burn-rate alerts against the SLO.
Trade-offs & common mistakes
Too many SLOs or internal-metric SLIs nobody cares about.
Aiming for 100% / setting SLO = current performance.
Alerting on thresholds (CPU) → fatigue; single-window burn alerts → slow reset or noise.
Tip: answer aloud for ~5 minutes before reading the solution.
Question
Your board and engineers want to "move to microservices" because the 8-year-old monolith slows delivery. As CTO, how do you decide whether to do it, how do you execute without a big-bang rewrite, and how do you record and communicate the decisions?
Step-by-step solution
Diagnose the real problem with data (lead time, deploy frequency, coupling hot spots, team contention) — microservices solve team/organizational scaling, not bad code.
If extracting: use the strangler fig — route via a facade, extract one capability at a time, start with a loosely coupled edge capability, then the high-value changing ones; decouple by domain (bounded contexts) and move the data with it.
Prerequisites: automated deploys, observability, on-call ownership per service.
Record decisions as ADRs (context, decision, consequences) in the repo; review significant ones openly.
Communicate in business terms: milestones, measured outcomes, explicit stop/continue checkpoints.
Final answer
Only split when data shows organizational/deployment bottlenecks; otherwise modularize. If splitting, do it incrementally with the strangler fig pattern by bounded context, data included, on top of solid platform prerequisites, and capture each decision in ADRs with measurable checkpoints.
Trade-offs & common mistakes
Big-bang rewrite; extracting services but sharing one database (distributed monolith).
Splitting by technical layer instead of business capability.