Picture a typical business-to-business software platform. It runs in three Availability Zones behind an Application Load Balancer, with Auto Scaling on every tier and a Multi-AZ database. It has survived instance failures, a zone outage, and a load test at four times peak. By every standard measure, it is highly available.
On a Tuesday afternoon, a configuration change goes out: a new validation rule with a regular expression that backtracks badly on one rare input. It reaches every worker in every zone within two minutes, because that is how configuration works. A few minutes later, one customer's nightly import sends that input. Every worker that picks up a request from that import pins a CPU and stops responding. The load balancer marks them unhealthy and moves their traffic to the others, which pick up the same requests from the same retrying client. Within ten minutes, all 2,000 customers are down.
This is a composite scenario, not one client's outage. Nothing in it is a hardware failure. Three zones did not help, because the defect was in all three.
Redundancy Is Not Isolation
Multi-AZ design protects against the failures that are independent across copies: a disk, a host, a rack, a data center. Each copy is the same, so when one copy's hardware fails, the others carry on.
The failures that cause the largest outages are not independent. A bad deploy, a configuration push, a poison request, a noisy tenant, a software limit reached by every host at the same time: each of these is present in every copy, because every copy runs the same code, gets the same configuration, and serves the same customers. More copies add more places for the same defect to run.
The public example is the Amazon Kinesis Data Streams event in US East (N. Virginia) on November 25, 2020. AWS's summary describes adding capacity to the Kinesis front-end fleet, which caused every server in that fleet to exceed the maximum number of threads its operating system configuration allowed. Every front-end server hit the same limit for the same reason. Among the remediations, AWS said it would greatly accelerate the cellularization of the front-end fleet to match what it had already done with the back end.
What limits that kind of failure is not more redundancy. It is isolation: making sure that no single change, request, or customer can reach everything at once.
Cells, Routers, and Size
A cell is a complete, independent copy of the workload: its own compute, its own data, its own queues and caches, and ideally its own AWS account. Cells share nothing on the request path. A customer, or a partition key such as a tenant or an account ID, is assigned to exactly one cell.
A thin router sits in front. Its only job is to look up the cell for a request and send it there. It holds no business logic, so it rarely changes, and it caches the mapping, so it keeps routing even if the mapping store is unavailable. The router is the one shared component, and it must be the simplest and most statically stable part of the system (post 8).
Four rules make cells work.
Cap cell size. A cell has a maximum number of customers or a maximum load, and it is tested at that size. When the business grows, you add cells, not capacity to existing cells. A cell's failure is then a known, bounded fraction of customers.
Deploy cell by cell. Every change, code or configuration, goes to one cell first, bakes there while alarms watch it, then proceeds in widening waves. An alarm stops the rollout automatically. The Builders' Library article Automating safe, hands-off deployments describes how AWS does this.
Keep cross-cell work off the request path. Global reports, billing, and search across all customers read from data each cell publishes, never by calling into cells during a request. A read model fed by every cell's events (post 7) is the usual answer.
Plan for moving customers. Cells fill unevenly, and a large customer can outgrow its cell. Migration between cells is an operation you build and rehearse, not one you invent during a crisis.
Shuffle sharding
Cells assign each customer to one group. Shuffle sharding goes further: it assigns each customer a small random set of workers drawn from a larger pool, so that two customers rarely share all of theirs.
The Builders' Library article Workload isolation using shuffle-sharding gives the small example. With eight workers and two per customer, there are 28 distinct pairs. A customer whose requests crash workers takes down its own two, and only a customer assigned the same pair loses all of its capacity. The rest still have at least one healthy worker. The count grows quickly with the pool:
combinations = n! / (k! x (n - k)!)
n = 8, k = 2 -> 28
n = 2,048, k = 4 -> 730,862,190,080
The second line is the scale of the same article's Amazon Route 53 example, where each customer domain is assigned four of 2,048 virtual name servers, about 730 billion possible shuffle shards. Shuffle sharding pays off when clients are fault tolerant and retry against the other members of their shard, so it fits fleets of interchangeable workers: name servers, API front ends, queue partitions.
Building It on AWS
On AWS, the strongest cell boundary is the account. An account per cell gives each cell its own service quotas, its own IAM boundary, and its own limit on how far a mistake can travel. That is why cells sit naturally on a multi-account foundation built with AWS Organizations, with each cell's account created from the same baseline.
| Service | Setting | Value | Source |
|---|---|---|---|
| AWS whitepaper | Reducing the Scope of Impact with Cell-Based Architecture | AWS's design guidance: cell sizing, routing, and migration | Whitepaper |
| AWS Organizations | One account per cell | Separate service quotas, IAM, and blast radius per cell | Organizations |
| Amazon DynamoDB or Amazon S3 | Cell map | Partition key to cell, cached by the router, which stays simple and keeps routing to healthy cells when one is unreachable | Cell routing |
| Amazon CloudFront | Router tier | CloudFront Functions with KeyValueStore choose the cell origin at the edge | CloudFront KeyValueStore |
| Amazon Route 53 | Router tier, DNS form | A DNS name per cell, with each customer pointed at its cell's name | Route 53 concepts |
| AWS CodePipeline or your CI | Wave deployment | One cell first, bake, then waves, stopping on an alarm | Automating safe, hands-off deployments |
A wave plan is simple to write down and hard to skip once it is in the pipeline:
waves:
- name: one-box # one cell, the smallest or an internal one
cells: [cell-07]
bake_minutes: 60
- name: wave-1
cells: [cell-01, cell-02]
bake_minutes: 30
- name: rest
cells: all-remaining
max_parallel: 3
stop_on: [cell_error_rate_alarm, cell_latency_p99_alarm]
What it costs
Cells are not free. You add a router and a mapping store, a fleet of accounts to baseline and keep consistent, cross-cell tooling for reporting and support, and a migration process. Each cell also carries fixed overhead, so very small cells waste money. Most teams start with a handful of large cells and split them as the business grows.
When Not to Use It
One tenant, or low traffic. If the workload serves one customer or a handful of internal users, there is no fraction to protect. A bad deploy reaches everyone either way. Invest in wave deployments within one stack and fast rollback instead.
No strict availability objective. Cells cost real money and operational effort. If the business can accept an hour of total downtime once a year, cheaper bulkheads cover most of the risk: separate connection pools per dependency (post 2), reserved concurrency per function, per-tenant throttles (post 4).
Data that cannot be partitioned. Cells need a partition key that keeps most requests inside one cell. A workload where every request touches every customer's data does not split cleanly, and forcing it creates a distributed system with all of the cost and none of the isolation.
How We Check for It
This sits in the Reliability pillar, under REL10, "How do you use fault isolation to protect your workload?", and its best practice REL10-BP03, "Use bulkhead architectures to limit scope of impact." Wave deployments belong to Operational Excellence, and the account-per-cell boundary to Security and to a sound multi-account foundation.
In a Vantalect review, we look at:
- What fraction of customers one bad deploy or configuration change can reach.
- Whether deployments proceed in waves with automatic stops on alarms, for configuration as well as code.
- Whether there are per-tenant limits, so one customer cannot consume shared capacity.
- Which components every request shares, and whether the router is the only one.
- Whether a cell, or its nearest equivalent, has ever been taken down on purpose to measure the impact.
The Well-Architected Review measures these in your live accounts.
Related Patterns
- Static Stability: the router must keep working without its control plane.
- Load Shedding and Backpressure: per-tenant limits are the first, cheaper bulkhead.
- Circuit Breakers: bulkheads inside a process, at the smallest scale.
- Observability That Answers Questions: error rate per cell is how you see a wave go wrong.
This post is part 9 of Cloud Design Patterns.