Retries Without Retry Storms:
Backoff, Jitter, and Idempotency Keys

Picture an order service that calls a payment API. The payment API has a bad minute. A deploy warms up, a cache is cold, or a dependency slows from 40 milliseconds to four seconds. Nothing is down. A fraction of calls simply time out.

Each timeout triggers a retry. The mobile app retries. The backend-for-frontend service retries behind it. The order service's SDK retries behind that. Three layers, three attempts each, and one tap on "Pay" becomes 27 calls against a service that was already struggling.

The payment API slows further. More calls time out, so more retries arrive. Within a minute the retry traffic is larger than the real traffic, and a dependency that had a bad minute now has a bad hour. Some customers see an error and tap "Pay" again. A few of them are charged twice.

This is a typical scenario, not a report about any one client. It is also how a retry storm works. No single layer did anything wrong. Each one did the sensible thing, and the sum was an outage.

A retry is a promise to add load at the worst possible moment. Before you make that promise, it needs a cap, full jitter, a budget, and an idempotency key. This post covers each one and what AWS gives you for them. It is the first of ten patterns in this series.

Why Retries Multiply

Retries are not free. Each one consumes a connection, a thread, and a slice of the server's capacity. When failures are rare and brief, retries are the right call. When failures come from overload, they are the one thing the server cannot afford.

There are three ways this goes wrong.

Layers multiply. The Amazon Builders' Library works through a call stack five layers deep with three retries at each layer. If every layer retries on its own, the load on the database rises 243 times. My three-layer example is the same arithmetic on a smaller stack: 3 × 3 × 3 is 27. Add a layer and the number triples.

Waves synchronize. If a thousand clients hit an error at the same moment and all wait 100 milliseconds, they all retry at the same moment. Backoff alone moves the spike later. It does not flatten it.

Writes duplicate. A timeout does not tell the client whether the server did the work. The response may have been lost after the charge went through. Retrying a write without a way to recognize the repeat creates a second order, a second charge, a second email.

AWS's own post-event summary shows the mechanism at a larger scale. In its summary of a DynamoDB disruption in US East on Sunday, September 20, 2015, storage servers could not retrieve membership data from a metadata service within the allowed time. The servers that failed kept retrying their requests, and the retries kept the metadata service under heavy load. At 5:06 AM PDT, AWS paused requests to the metadata service to reduce retry activity, then added capacity. Most customers saw low error rates again at 7:10 AM PDT. That was AWS's internal retry traffic, not a customer's, but the mechanism is the same.

How the Pattern Works

Five rules make a retry safe. Together they keep it small, spread out, and harmless to repeat.

1. Retry only what can succeed. Retry timeouts, connection resets, HTTP 5xx, and throttling errors. Never retry a validation error or an access-denied error. They will fail the same way every time.
2. Retry at one layer. Pick the layer closest to the failure and make every other layer fail fast. REL05-BP03 names retrying at multiple layers as an anti-pattern for this reason.
3. Back off with full jitter. Each wait is a random value between zero and a ceiling that doubles with each attempt up to a cap (the formula follows this list). The random multiplier spreads a crowd of clients across the whole window instead of stacking them at its edge.
4. Spend from a budget. A client-side retry budget lets retries proceed while most calls succeed and stops them when failures become widespread. Failing fast during an outage frees threads and connections, and it gives the dependency room to recover.
5. Make the write safe to repeat. The client generates a unique key for each logical operation and sends the same key on every attempt. The server records the key with the result. A repeat gets the stored result back instead of a second execution.

The backoff in rule 3, written out:

delay = random(0, 1) × min(cap, base × 2^attempt)

Two details matter as much as the five rules. Set a timeout shorter than your caller's timeout, so a slow call is abandoned while its caller is still waiting. And store the idempotency record in the same transaction as the work, so the service never keeps a key without its result or a result without its key.

The diagram shows the last rule at work: the first response is lost, the client waits a random interval, and the retry returns the stored order.

Building It on AWS

The AWS SDKs implement most of this for you, which is the reason to use them instead of writing your own. One caveat matters today. As of October 18, 2026, the SDK retry behavior described below is opt-in: set AWS_NEW_RETRIES_2026=true to get it. AWS's announcement says it becomes the default in November 2026. Without the setting, your SDK uses its pre-2026 behavior, which differs by language in backoff timing, retry quota costs, and service-specific defaults. The announcement says base delays ranged from 10 milliseconds to 1,000 milliseconds across SDKs, and the transient-error retry cost was 5 tokens before it became 14. So the first job is to find out which behavior your clients run today. Every number in the table is the 2026 behavior unless it says otherwise.

Step Functions is the trap in the table. Its retrier does not add jitter unless you ask. The default is NONE, so every execution that fails together retries together.

"Retry": [{
  "ErrorEquals": ["States.TaskFailed"],
  "IntervalSeconds": 2,
  "MaxAttempts": 3,
  "BackoffRate": 2.0,
  "MaxDelaySeconds": 20,
  "JitterStrategy": "FULL"
}]

For the key itself, the Builders' Library article on making retries safe with idempotent APIs describes the contract: the same key returns an equivalent response, and the same key with different parameters returns a validation error.

When Not to Use It

Retries are the wrong tool in four cases.

• The call mutates state and has no key. A retried "create order" without an idempotency key is a duplicate order waiting to happen. Add the key first. If you cannot change the API, do not retry the write. Surface the error and let the user decide.
• Every layer already retries. If the SDK, the service, and the caller each retry, you have built the 27-call stack. Keep retries at one layer, usually the one closest to the failing call, and set max_attempts to 1 everywhere else.
• The dependency says it is overloaded and your deadline is nearly gone. A retry that cannot finish before the caller gives up is pure waste. Check the remaining time budget before each attempt, and fail fast when it is shorter than the next wait. When the failure persists, stop retrying and open a circuit breaker, the subject of the next post.
• One client serves many tenants and you are tempted by adaptive mode. AWS says to avoid it there: throttling on one resource slows requests to every other resource on the same client, including the healthy ones. Use standard mode and give each tenant or resource its own client if you need isolation.

How We Check for It

This pattern sits in the Reliability pillar of the AWS Well-Architected Framework, under REL05 ("How do you design interactions in a distributed system to mitigate or withstand failures?") and REL04 ("How do you design interactions in a distributed system to prevent failures?"). The best practices are REL05-BP03, control and limit retry calls, REL05-BP05, set client timeouts, and REL04-BP04, make mutating operations idempotent.

In a live account, we look at:

• SDK client configuration in code and environment: the retry mode, max_attempts, and whether AWS_NEW_RETRIES_2026 is set.
• Timeouts on every remote call, compared with the caller's own timeout.
• The number of retrying layers on one request path, counted from the client to the data store.
• Idempotency keys on every mutating API, and a TTL on the records that store them.
• Retry counts and attempt spans in X-Ray traces, to see what a failure costs in real calls.

If you want this check run against your own accounts, it is part of the AWS Well-Architected Framework Review.

Related Patterns

• Circuit Breakers: where retries stop and the breaker starts.
• Load Shedding and Backpressure: a throttling error is the server asking you to slow down.
• Event Fan-Out With Dead-Letter Queues: queues redeliver messages, so every consumer must be idempotent.
• Observability With RED, USE, Traces, and SLOs: you cannot cap what you do not count.