Queue-Based Load Leveling:
Let the Queue Take the Spike

Here is how this usually goes wrong. Picture a course enrollment service. On an ordinary Monday it takes 100 requests a second, and each request writes a registration row to a PostgreSQL database. Everything is synchronous: the API validates the request, writes the row, and answers.

Then registration opens for the fall term. For ten minutes the service takes 2,000 requests a second.

The stateless API tier does what it was built to do and scales out. That is the problem. Every new task opens its own database connections, the database hits its connection limit, and writes start to queue. Requests time out at the load balancer. Browsers retry. The retries add load to a database that was already the bottleneck, and the incident ends when the spike does, not when anyone fixes anything.

That is a composite, not one client's outage. Nothing in it is a bug. The API tier scaled, the database held its ground, and the two together produced an outage because the system's peak capacity was the database's peak capacity.

Peak Capacity Is the Wrong Target

A synchronous request ties the caller to everything downstream. The user waits while the API waits while the database works. Whatever the slowest shared dependency can do per second is the most the whole system can do per second, no matter how many API tasks sit in front of it.

Auto scaling does not fix this. It fixes the stateless tier, and the stateless tier was never the constraint. Adding tasks in front of a database moves the queue from the load balancer into the connection pool, where it is harder to see and more expensive to leave.

You have two choices. Size the database for the peak and pay for that capacity every other minute of the year. Or ask whether the work needs an answer in the same request.

Often it does not. A registration, a receipt, a thumbnail, a report, a notification: the user needs to know the request was accepted, not that every side effect has finished. The AWS Well-Architected Framework says this directly in REL04-BP02, Implement loosely coupled dependencies: make interactions asynchronous where possible, for any interaction that does not need an immediate response.

The pattern, then, is simple to state. Decouple acceptance from execution. Take the spike at the door, and do the work at the rate the database can sustain.

How the Pattern Works

The API validates the request and puts a message on a durable queue. It returns 202 Accepted with a status location the client can poll. That response takes milliseconds and touches nothing slow, so the front door stays fast at any arrival rate.

Behind the queue, a pool of consumers pulls messages and does the work. The pool has a hard cap on concurrency, chosen from what the database can handle, not from how fast the messages arrive. The queue absorbs the difference. During the spike it grows. After the spike it drains.

Three details make the pattern safe instead of merely hopeful.

First, the cap. Without one, the consumers scale to meet the backlog and recreate the original outage one layer down. The cap is the part of the pattern that protects the database.

Second, the clock. A queue that never drains is its own outage. The Amazon Builders' Library article Avoiding insurmountable queue backlogs describes systems that run in a fast mode with no backlog and a slow mode once arrivals outrun processing. Its advice is to watch message age, not just depth, because depth does not tell you how long anyone has been waiting.

Third, the failure path. A message that cannot be processed must not block the ones behind it forever, so it moves to a dead-letter queue after a bounded number of tries, and a human looks at it.

Little's Law Sets the Expectation

Little's law says the average number of items in a system equals the arrival rate times the average time each item spends there: L = λW. It works for concurrency and for backlog, and it tells you whether the queue will drain.

Take the enrollment service. Each message needs 80 milliseconds of database time, and the database is comfortable with 40 concurrent connections. At 2,000 messages a second, the synchronous path needs 2,000 × 0.08 = 160 connections, four times what the database is happy with. That is the outage.

With a queue and a cap of 40 consumers, throughput is 40 ÷ 0.08 = 500 messages a second. During the ten-minute spike the backlog grows by 1,500 a second, to 900,000 messages. When arrivals fall back to 100 a second, the queue drains at a net 400 a second and clears in 2,250 seconds, about 37 minutes. The last message of the spike waits about 30 minutes for its turn.

That is the trade. Nobody got an error, and the database ran at its comfortable limit instead of its breaking point. The cost was a 30-minute worst-case delay, so the question to ask first is whether the business can live with it. If average arrivals ever exceed 500 a second, the backlog never drains at all. The queue only delays that failure.

The numbers are a model, not a measurement: they ignore batching and assume every message takes the same time.

Building It on AWS

On AWS, the usual shape is an Amazon SQS queue feeding an AWS Lambda event source mapping. The settings that matter are the cap on consumers, the visibility timeout, the retention period, and the alarm. The defaults are not the safe values for all four. Out of the box, Lambda scales an SQS event source mapping from 5 concurrent invocations by up to 300 more each minute, to a maximum of 1,250, which is far more than most databases can serve (Lambda scaling behavior).

AWS describes the throughput of a standard queue as nearly unlimited per API action, so the queue is rarely the limit. The consumer is. With a 30-second function timeout, set the queue's visibility timeout to at least 180 seconds. If the function fails, a message reappears after that timeout, so every consumer must tolerate seeing the same message twice, which is the subject of Retries Without Retry Storms.

aws lambda create-event-source-mapping \
  --function-name enrollment-writer \
  --event-source-arn arn:aws:sqs:us-east-1:111122223333:enrollments \
  --batch-size 10 \
  --function-response-types ReportBatchItemFailures \
  --scaling-config '{"MaximumConcurrency":40}'

When Not to Use It

The user needs the answer in this request. A price quote, a login, a seat check: the caller cannot proceed without the result, and a 202 is a lie. Keep these synchronous and protect them directly, with a cache, a read replica, or the techniques in Load Shedding and Backpressure.

Average arrivals exceed average capacity. A queue smooths a spike. It cannot fix a deficit. If the model above says the backlog never drains, add capacity or reduce the work, because a queue here only turns a visible failure into a delayed one.

Strict ordering across all messages matters. FIFO queues order messages within a message group, and by default each partition is limited to 300 transactions per second per API action, far below what a standard queue offers. When one global order is a real requirement, that ceiling is probably too low for a spike. Partition by an entity, such as an account, so ordering holds only where it must.

The work is too cheap to queue. If handling a request costs less than putting it on the queue and taking it off, the queue adds latency, a second failure mode, and a bill. Do the work inline.

How We Check for It

This sits in the Reliability pillar, under REL04 and its best practice REL04-BP02, "Implement loosely coupled dependencies," and under REL07, "Design your workload to adapt to changes in demand." The Performance Efficiency pillar covers matching capacity to demand. In a live account, a Vantalect review looks at:

• Which write paths are synchronous that need not be, found by tracing what the user waits for against what they need.
• The consumer concurrency cap against the database's connection limit, including any proxy in between.
• The queue's visibility timeout against the consumer's timeout, and whether it covers the six-times guidance.
• A dead-letter queue on every source queue, with longer retention than the source and an alarm on its depth.
• An alarm on ApproximateAgeOfOldestMessage, with a threshold tied to a business deadline and a runbook that someone has actually read.

To see how we run these checks across a whole account, read about the Well-Architected Review.

Related Patterns

• Load Shedding and Backpressure: for when the queue itself must say no, because the backlog has grown past usefulness.
• Event Fan-Out With Dead-Letter Queues: for sending one event to many consumers and handling the messages that fail.
• Retries Without Retry Storms: for making consumers safe against the duplicate deliveries a queue produces.
• Observability With RED, USE, Traces, and SLOs: for turning the age-of-oldest-message alarm into an objective.

The full set is on the Cloud Design Patterns page.