Idempotency Keys: A Pattern Every Senior Engineer Should Master

What you need to know

• Idempotency keys convert unsafe-to-retry operations into safe-to-retry ones. This is the design goal, and everything about the implementation should serve it.
• The key must be generated before the first request, stored by the client, and reused for all retries. A key generated anew for each retry does not provide idempotency.
• The server-side implementation requires a persistent store for key-to-result mappings and a TTL. In-memory caching is insufficient because server restarts lose the stored keys.
• The partial execution edge case is the hardest part of the implementation. The server must handle the case where it received the key, began processing, failed mid-execution, and now receives the same key again.
• Idempotency keys are a server-side implementation that requires client-side cooperation. Document the expected behavior clearly in the API reference.

The core argument

The pattern that most engineers implement first is the happy path: key comes in, check the store, if absent process and store, if present return the stored result. This works for the common case. The senior engineer distinction is in handling the edge cases: the in-flight request where the server started processing but has not yet stored the result, the race condition where two requests with the same key arrive simultaneously, and the failure where the operation was stored as successful but the response was lost in transit.

The in-flight case requires a two-phase commit pattern: mark the key as processing before executing the operation, mark it as completed with the result after. If a request arrives for a key that is in the processing state, the server should either wait for the first request to complete or return a 409 Conflict that tells the client the operation is in progress. The race condition case requires a database-level unique constraint on the key column, not just an application-level check, to prevent simultaneous processing under concurrent requests.

The lost response case is often overlooked. The operation completed successfully and the result was stored against the key, but the response never reached the client. The client retries. The server returns the stored result. This is the intended behavior. The client must be designed to handle receiving the same response twice, which means the response body should contain enough information for the client to determine that the operation was already completed. Returning the resource ID and state in every mutation response makes this straightforward.

Common mistakes

1. Using in-memory storage for key results. In-memory storage is lost on server restart. A server restart during a high-traffic period means all in-flight idempotency keys become orphaned and retries execute as fresh operations.

1. Not handling the partial execution case. An implementation that checks for keys but does not handle the case where execution started and failed will process the operation twice when the same key arrives after a mid-execution failure.

1. Using timestamps as part of the key. A timestamp-based key is not safe for retries that happen within the timestamp resolution. Use UUIDs.

1. Setting the TTL based on convenience rather than client behavior. The TTL must be longer than the maximum realistic retry window. If clients retry for up to six hours after a network partition, a two-hour TTL is insufficient.

1. Not including idempotency key support in the API changelog. Adding or changing idempotency key behavior is a breaking change for clients that have built around the previous behavior. Document it and version it appropriately.

Where to start

1. Implement the database schema first. A table with key, status, result, created_at, and expires_at. Add a unique constraint on key. This schema serves the full pattern including the partial execution case.

1. Write the middleware that handles the key check. Before the handler executes: check for existing key. If found and completed, return stored result. If found and processing, return 409. If absent, mark as processing and continue.

1. Write the handler completion logic. After the handler executes successfully: update the key status to completed and store the response body. Run this in a transaction with the primary operation.

Related reading

• Idempotency in API Design: Why It Matters More Than You Think
• Building APIs That Survive Five Years of Customer Change Requests
• SaaS Webhook Reliability: From At-Most-Once to At-Least-Once to Exactly-Once
• Resilience Patterns: Circuit Breakers, Retries, Bulkheads