Load Shedding and Backpressure:
Saying No Early Keeps the Lights On

Here is how this usually goes wrong. Picture an order service that can complete 400 requests a second. A marketing email goes out, and traffic climbs to 600. Nothing crashes. CPU sits at 70 percent, the health checks pass, and the dashboards stay green.

The extra 200 requests a second go into queues: the load balancer's, the web server's, the thread pool's. Each request waits a little longer than the one before it. Median latency passes two seconds, then three. The clients time out at three seconds and retry.

Now the service does the same work twice. It finishes a request, writes the response, and the caller is already gone. The retry joins the back of the queue. Offered load is no longer 600 requests a second. It is closer to 900, and every answer arrives after the deadline it was meant for.

The machines are busy and the error rate on the server looks low. Almost nothing the service does is useful. That is congestive collapse, and no hardware failed to cause it. The service accepted work it could not finish in time.

An overloaded service that tries to serve everyone serves no one. The fix is to reject the excess early, cheaply, and on purpose, and to tell callers to slow down.

Throughput Is Not Goodput

Throughput is every request the server handles. Goodput is the part of it that finishes without an error and fast enough for the client to use the response. The Amazon Builders' Library article on using load shedding to avoid overload draws exactly this line. Past saturation, throughput stays high while goodput falls toward zero.

Queues hide the problem: a queue adds delay to every request in it while the service looks healthy. The article points out that when median latency equals the client timeout, about half of requests time out, so a latency problem has become an availability problem.

Two things make it worse:

• Wasted work. A request that waited past its caller's timeout still costs the full processing time. The answer goes nowhere.
• Retries. Each timeout becomes a new request. The article calls out this feedback loop: timed-out work wastes server effort, and the retries multiply the load.

AWS has described this failure in its own infrastructure. In its summary of the October 19 and 20, 2025 service disruption in US East (N. Virginia), a DynamoDB DNS failure caused the EC2 subsystem that manages leases on physical hosts to lose its leases. When DynamoDB's DNS was restored at 2:25 AM PDT, the subsystem tried to re-establish them. The summary says the work could not finish before it timed out, and more work was queued to try again. In its words, the subsystem "had entered a state of congestive collapse."

The way out was throttling. At 4:14 AM PDT, engineers throttled incoming work and restarted hosts selectively, which cleared the queues. By 5:28 AM the leases were back, though many requests still returned "request limit exceeded" because of that throttling. Engineers began relaxing the throttles at 11:23 AM, and EC2 was operating normally at 1:50 PM. I read that as a lesson about the price of the cure: shedding works, and it hurts, so decide where it hurts before the incident.

How Load Shedding Works

The pattern has four parts. Each one removes a way for an overloaded service to waste work.

Bound every queue. An unbounded queue is a promise to be late. Cap its size and its age, and reject new work when the cap is hit. The Builders' Library article recommends bounding both and dropping requests that have waited too long.

Admit by concurrency, not CPU. CPU is a lagging signal and shedding keeps it low. The number of requests in flight, or the time a request spent waiting, tells you the service is full. Set the limit from a load test past saturation, not from a guess.

Reject cheaply, before the expensive work. A rejection that costs as much as the work saves nothing. Check the limit first, before parsing a large body, opening a connection, or calling a dependency. Return a 429 or 503 and say how long to wait. The article notes that a fast rejection is generally the safe default over letting a surge queue build.

Drop work nobody is waiting for. Pass the caller's deadline down every hop. When a request arrives with its deadline already past, drop it without processing.

Then decide who goes first. Not all traffic is equal. The article gives examples: load balancer health checks come first, because dropping them can shrink your fleet. A request that finishes an in-progress operation outranks one that starts a new one. Human traffic outranks crawlers. Shed the low-priority work first and keep the rest inside its latency budget.

The last part is backpressure. A rejection is a message to the caller. Callers that read it must slow down, with exponential backoff, jitter, and a retry budget, as in post 1 of this series. Without that, shedding invites a retry storm against the very layer that is protecting you.

Building It on AWS

Layer the controls, outermost first. AWS WAF limits one source before it reaches your API, and API Gateway throttles the account, the stage, and each API key. Inside the service, cap concurrency yourself, and use Lambda's reserved concurrency as a hard ceiling in front of a fragile dependency. Check AWS's numbers before you rely on them: the API Gateway defaults are lower in some Regions, and its throttles are targets, not guarantees.

Inside the service, the admission check is a few lines. This sketch assumes one asyncio event loop, so the counter needs no lock:

MAX_INFLIGHT = 64            # set from a load test past saturation
inflight = 0

async def handle(request):
    global inflight
    if request.deadline_passed():              # the caller has given up
        return reject(503)
    if inflight >= MAX_INFLIGHT:               # full: say no before any work
        return reject(429, retry_after=1)
    inflight += 1
    try:
        return await process(request)
    finally:
        inflight -= 1

When Not to Use It

Load shedding is a guard, not a capacity plan.

Nobody watches the rejection rate. Shedding hides a capacity problem as quiet 429s. If rejections run all day, the service is undersized. Alarm on the rejection count and the rate, and fix capacity.

The work cannot be retried safely. Do not shed a payment or a state change unless the caller can tell it was never applied and retry it safely. Make the operation idempotent first (post 1), or put the request on a durable queue so it is accepted once.

One caller, steady rate. An internal batch job that writes to one database needs a bounded worker pool or a queue, not an edge throttle. Level the load with a queue and cap the consumers.

Throttles you never tested. A limit set at the default and never load tested is a guess. Test at the limit before an incident does.

How We Check for It

Pillar: Reliability. Question: REL05, on designing the interactions in a distributed system to mitigate or withstand failures. Best practices: REL05-BP02 throttle requests, REL05-BP04 fail fast and limit queues, and REL05-BP07 implement emergency levers. In a Well-Architected Review, we look at:

• Throttles at the edge (WAF rate-based rules, API Gateway limits) and per client (usage plans), with values someone chose on purpose.
• Bounded queues and concurrency limits in the service, and Lambda reserved concurrency on functions that front a database.
• A load test past saturation whose report shows goodput holding steady, not just peak requests per second.
• Rejection counts and 429 and 503 rates on the service dashboard, with an alarm.
• A written emergency lever, such as a flag that sheds low-priority traffic, that someone has tried.

The review itself is described on our Well-Architected page.

Related Patterns

This is post 4 of the Cloud Design Patterns series.

• Retries Without Retry Storms: the caller's side of backpressure, with backoff and a retry budget.
• Queue-Based Load Leveling: when work can wait, a queue absorbs the spike instead of rejecting it.
• Cell-Based Architecture: cells cap how much load any one failure can see.
• Observability With RED, USE, Traces, and SLOs: the rejection rate and goodput are the signals that tell you shedding is working.