Picture a typical web tier spread across three Availability Zones. Six instances, two per zone, run at about 70% CPU at the daily peak. The capacity plan is sized to the peak, because Auto Scaling will add more if it is ever needed. The disaster recovery runbook says the same thing in more words: if a zone fails, Auto Scaling replaces the lost instances in the other two.
Then a zone fails during a larger event, one that also slows the APIs that launch instances. The two lost instances' traffic lands on the remaining four, which are now at about 105% of their capacity. Auto Scaling sees the load and tries to launch more. The launches queue, or fail, or come up after the event is over. For the next two hours, the surviving instances carry half again their normal peak, more than they can serve, latency climbs, and the health checks start removing instances that are only slow.
This is a composite scenario, not one client's outage. The design worked exactly as specified. The specification assumed that recovery could depend on launching something at the moment launching was hardest.
Recovery That Depends on Acting
Most recovery plans are plans to do something. Scale out in the surviving zones. Update a DNS record. Promote a replica from the console. Each of those is a call to a control plane: the part of a service that creates, changes, and deletes resources. The part that keeps running what already exists, serving requests, answering DNS queries, routing packets, is the data plane.
AWS's own guidance is direct about the difference. The AWS Fault Isolation Boundaries whitepaper describes control planes as complicated orchestration systems with more moving parts, which makes failures statistically more likely there than in the data plane. The Builders' Library article Static stability using Availability Zones and the Well-Architected best practice REL11-BP04 draw the conclusion: during recovery, depend on the data plane.
There is a second, less obvious problem. In a large event, everyone's recovery plans run at once. Every customer in the Region whose plan says "launch more" calls the same launch APIs in the same minutes.
The public example is the AWS service disruption in US East (N. Virginia) on October 19 and 20, 2025. AWS's summary describes new EC2 instance launches failing or impaired for hours after the initial DNS problem with DynamoDB was fixed, while instances that were already running were not affected. It also describes Network Load Balancer health checks failing for new instances brought into service before their network state had propagated, which pulled load balancer nodes out of DNS and triggered automatic zone failovers until engineers turned them off. A workload whose recovery needed new instances that day had to wait. A workload whose capacity was already running did not need a launch to recover.
Static Stability
A system is statically stable when it keeps working through an expected failure without making any change at all. Nothing launches, nothing scales, nothing is reconfigured. It is the difference between a building that has a backup generator already running and one that plans to buy a generator when the power fails.
Four techniques get you there.
Pre-provision for the loss of a zone. If a workload runs in N zones and must survive losing one, each zone must be able to carry peak divided by N minus 1. The arithmetic:
capacity per zone = peak / (N - 1)
total provisioned = peak x N / (N - 1)
N = 2 -> 200% of peak (each zone runs at 50% at peak)
N = 3 -> 150% of peak (each zone runs at 67% at peak)
N = 4 -> 133% of peak (each zone runs at 75% at peak)
Auto Scaling still has a job. It adds capacity for growth and removes it at night. It is not the recovery mechanism.
Prefer data-plane mechanisms that are already in place. Health checks, load balancer routing, standby replicas that already exist, Route 53 failover records that already point at both targets. They act without anyone calling an API.
Make failover a pre-armed switch, not a provisioning step. The switch moves traffic. It does not create anything. Amazon Application Recovery Controller's zonal shift is the cleanest example: it moves traffic for a load balancer, an Auto Scaling group, or an Amazon EKS cluster away from a zone without changing the workload's configuration.
Keep last-known-good configuration. If a service reads feature flags, routing tables, or secrets from somewhere at startup or on a timer, it must keep running on the values it already has when that source is unreachable. A configuration fetch that fails closed turns a minor dependency into an outage.
Building It on AWS
The setup is mostly settings you already have, sized and tested differently.
| Service | Setting | Value | Source |
|---|---|---|---|
| Amazon EC2 Auto Scaling | Minimum capacity across three zones | Sized so two zones carry peak; scaling is for efficiency, not recovery | Static stability using Availability Zones |
| Amazon Application Recovery Controller | Zonal shift and zonal autoshift | Moves traffic away from an impaired zone for Application and Network Load Balancers, Auto Scaling groups, and Amazon EKS | Zonal shift, supported resources |
| Amazon Route 53 | Health checks and failover records | Data-plane failover; avoid editing records during an event | Route 53 concepts |
| Elastic Load Balancing | Cross-zone load balancing | On by default for Application Load Balancers, off by default for Network Load Balancers; know which before an event | How Elastic Load Balancing works |
| Amazon Aurora | Replicas in each zone | Failover promotes a replica that already exists; service is typically restored in under 60 seconds | Aurora high availability |
| AWS Fault Injection Service | AZ Availability: Power Interruption scenario | Proves the design instead of assuming it | FIS AZ availability scenario |
Three details are worth knowing before the event, not during it.
Aurora replicas need to exist. A cluster with no replica recovers by creating a new primary instance, which is a control-plane action, instead of promoting one that is already running. Put at least one replica in a different zone from the writer.
Credentials and configuration should already be cached. The AWS SDKs cache temporary credentials and refresh them before they expire. Do the same for application configuration: AWS AppConfig Agent, for example, serves configuration from a local cache and polls for updates in the background, so the application does not need the service to answer every read.
Practice the switch. Zonal autoshift requires practice runs, in which AWS shifts traffic away from a zone on a schedule to prove the remaining zones can carry it. That requirement is the whole point: a static-stability claim that has never been tested is a guess.
What it costs
Static stability costs idle capacity, a third of the fleet in a three-zone design, and the bill makes that visible every month. Two things soften it. The baseline is predictable, so it is the cheapest capacity to cover with Savings Plans or Reserved Instances. And not every tier needs it: a stateless web tier and its database do, a reporting job that can wait an hour does not.
The alternative is to choose a degraded mode deliberately. A service that sheds low-priority work (post 4) and serves cached or stale data (post 5) can run at less than full capacity through a zone loss without falling over. That is also a static design, as long as the degraded mode needs no one to act.
When Not to Use It
The workload can wait. A batch system, a nightly report, or an internal tool used during business hours can wait out an outage and catch up. Paying 50% more for capacity that rides out a zone loss buys nothing it needs. Make sure the jobs can resume, and stop there.
The managed service already does it. Amazon S3 and DynamoDB are designed to keep serving through the loss of a zone within a Region. Building your own zone redundancy on top of them is cost without benefit. Spend the effort on the tiers you run yourself.
The workload has no availability objective. If nobody has written down how long the service may be down, decide that first. Static stability is a means to a stated objective, not a virtue on its own.
How We Check for It
This sits in the Reliability pillar, under REL11, "How do you design your workload to withstand component failures?", and two of its best practices: REL11-BP04, "Rely on the data plane and not the control plane during recovery," and REL11-BP05, "Use static stability to prevent bimodal behavior." It also draws on REL10-BP01, "Deploy the workload to multiple locations."
In a Vantalect review, we look at:
- Per-zone capacity against measured peak, for every tier on the critical path.
- The recovery runbook, with each step marked as a data-plane or a control-plane action.
- Whether zonal shift or zonal autoshift is configured for the load balancers in front of the workload.
- Whether each database has a replica in a second zone.
- The result and date of the last zone-failure test, if there has been one.
The Well-Architected Review checks all five against your live accounts.
Related Patterns
- Cell-Based Architecture: static stability contains a zone failure; cells contain your own mistakes.
- Circuit Breakers: a good fallback is one that does not call a control plane.
- Cache-Aside With Request Coalescing: last-known-good data is a static-stability tool.
- Observability That Answers Questions: per-zone capacity and health belong on the dashboard.
This post is part 8 of Cloud Design Patterns.