Event Fan-Out With Dead-Letter Queues:
Publish Once, Fail Alone

Picture an order service with five consumers. When an order is placed, it has to charge the card, send a confirmation email, reserve stock, update the loyalty balance, and feed the analytics pipeline. This is a labeled hypothetical, but the shape is common.

In the first version, the order service calls all five in sequence. The email provider has a bad afternoon, the fourth call times out, and the order fails, after the card was already charged.

In the second version, the order service writes one message to a shared queue, and a single worker does all five jobs. Then a customer enters a shipping name the email template cannot render. The worker throws, the message becomes visible again after the visibility timeout, and the worker throws again. It is one message, but it fails the whole batch it arrives in, so good orders behind it are retried too, and some customers get two confirmation emails.

In the third version, somebody adds a dead-letter queue, and the poison message finally moves out of the way. Nobody alarms on it. Three weeks later, finance asks why 212 orders never earned loyalty points. The dead-letter queue kept those messages only as long as its retention period allowed, and most of them are gone.

One Producer, Many Fates

The problem underneath all three versions is coupling. When a producer calls each consumer directly, the producer's availability is the product of every consumer's availability, and its latency is the sum of theirs. Five dependencies at 99.9% each give a call path that succeeds about 99.5% of the time, before anything else goes wrong.

Putting everything behind one queue removes the latency coupling but keeps the failure coupling. A single consumer that does five jobs retries all five when one fails. Splitting it into five workers reading one queue does not help either: a queue delivers each message to one receiver, so five different consumers on one queue compete for messages instead of each getting a copy.

Two more problems hide in every asynchronous design.

Some messages will never succeed. A malformed payload, a reference to a deleted record, or a bug in one consumer's code turns a message into a poison message. Without a limit, it is retried until the queue's retention period expires, consuming capacity the whole time.

Delivery is at least once. Amazon SQS standard queues and Amazon SNS standard topics deliver at least once, and Amazon EventBridge retries delivery to its targets, so every consumer will eventually see a duplicate. A consumer that charges a card or sends an email on every delivery will do it twice.

Publish Once, Queue Per Consumer

The pattern has five parts.

Publish each event once, to an SNS topic or an EventBridge event bus. The producer's job ends when the event is accepted. It does not know or care how many consumers exist, which means adding a sixth consumer is a subscription, not a change to the order service.

Give each consumer its own SQS queue, subscribed to the topic or targeted by a rule, with a filter so it receives only the events it needs. Each queue scales, retries, and fails independently. When the email consumer is down, its queue grows and nothing else notices.

Bound the retries. On the source queue, a redrive policy with maxReceiveCount moves a message to a dead-letter queue after that many failed receives. Five is a reasonable starting point: enough to ride out a brief dependency failure, few enough that a poison message leaves quickly. With AWS Lambda as the consumer, turn on partial batch responses (ReportBatchItemFailures) so one bad message does not send nine good ones back to the queue.

Treat every dead-letter queue as an incident queue. Alarm when it holds any message at all. Give it an owner who is paged, a runbook, and a retention period longer than the source queue's, because for standard queues SQS keeps a message's original enqueue time when it moves to the dead-letter queue. A message that spent three days in a four-day source queue has one day left unless the dead-letter queue keeps messages longer. Once the cause is fixed, redrive the messages back to the source queue.

Make consumers idempotent. Record a processed event ID with a conditional write, and skip events you have already handled. Post 1 covers idempotency keys in detail.

SNS or EventBridge

Both work. SNS is the simpler, higher-throughput fan-out: one topic, many SQS subscriptions, filter policies on message attributes or the payload, and FIFO topics when order matters within a group. EventBridge routes on the content of the event with rules, reaches more kinds of targets directly (Step Functions, API destinations, other buses and accounts), and can archive events and replay them after a consumer fix. When the events are business facts that several teams will want to subscribe to over the years, I lean toward EventBridge. When the job is to copy a high-volume stream to a handful of queues, SNS.

Two kinds of dead-letter queue

There are two places a message can die, and each needs its own dead-letter queue. A delivery failure happens when SNS or EventBridge cannot hand the message to the target at all: the queue was deleted, or a change to its access policy blocks the service. That dead-letter queue is set on the SNS subscription or the EventBridge target. A processing failure happens when the consumer receives the message and cannot handle it. That dead-letter queue is the SQS redrive policy. A common mistake is to configure one and assume it covers both.

Building It on AWS

Every value below comes from the AWS documentation linked in the last column. Check the EventBridge retry defaults for the kind of bus you use; they differ.

The queue, its dead-letter queue, and the alarm belong together in one template, so nobody can create the first without the other two. In AWS CloudFormation:

EmailQueue:
  Type: AWS::SQS::Queue
  Properties:
    VisibilityTimeout: 180            # six times a 30-second function timeout
    RedrivePolicy: { deadLetterTargetArn: !GetAtt EmailDLQ.Arn, maxReceiveCount: 5 }
EmailDLQ:
  Type: AWS::SQS::Queue
  Properties: { MessageRetentionPeriod: 1209600 }   # 14 days, the maximum
EmailDLQAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    AlarmActions: [!Ref OnCallTopic]  # an SNS topic that pages the owner
    Namespace: AWS/SQS
    MetricName: ApproximateNumberOfMessagesVisible
    Dimensions: [{ Name: QueueName, Value: !GetAtt EmailDLQ.QueueName }]
    Statistic: Maximum
    Period: 60
    EvaluationPeriods: 1
    Threshold: 0
    ComparisonOperator: GreaterThanThreshold

When Not to Use It

The caller needs every consumer's answer. If checkout cannot confirm until payment and inventory both succeed, fan-out is the wrong shape. Use a synchronous call for the step that must succeed, or an AWS Step Functions workflow that waits for both and compensates on failure.

There is one consumer. A topic in front of a single queue adds a hop and a policy to maintain. Send to the queue directly (post 3) and add the topic the day a second consumer appears.

Events must be processed in one global order. Fan-out multiplies an ordering problem by the number of consumers. If order matters only per customer or per order, use FIFO topics and queues with a message group per key. If it truly matters globally, rethink whether these should be events at all.

How We Check for It

This sits in the Reliability pillar, under REL04, "How do you design interactions in a distributed system to prevent failures?", and its best practice REL04-BP02, "Implement loosely coupled dependencies," and under REL06, "How do you monitor workload resources?" The redrive runbook belongs to Operational Excellence.

In a Vantalect review, we look at:

• Whether every SNS subscription, EventBridge target, and SQS queue that feeds a consumer has a dead-letter queue.
• Whether every dead-letter queue has an alarm, an owner, and a retention period longer than its source queue's.
• Whether consumers are idempotent, and how they know an event is a duplicate.
• Whether a redrive has ever been rehearsed, and how long it took.
• Whether any producer still calls its consumers directly on the request path.

The Well-Architected Review runs these checks against your live accounts.

Related Patterns

• Retries Without Retry Storms: redelivery makes idempotent consumers mandatory.
• Queue-Based Load Leveling: each consumer's queue is also its shock absorber.
• CQRS With Read Models: read models are often just another subscriber to the same events.
• Observability That Answers Questions: dead-letter queue depth is one of the signals worth paging on.

This post is part 6 of Cloud Design Patterns.