Cache-Aside With Request Coalescing:
Surviving the Cold Cache

Picture a typical retail catalog on AWS. The product pages read from an Amazon ElastiCache cluster, and the cache answers 98% of reads. The Amazon Aurora database behind it is sized for the other 2%, and on a normal weekday it idles at 15% CPU.

Then a cache node is replaced during a maintenance window. Or a deploy changes the key prefix. Or someone flushes the cluster to clear one bad entry. For the next few minutes, every product view is a miss, and a database sized for 2% of reads receives 100% of them: fifty times its normal read load, all at once, much of it for the same few hundred popular rows.

Connections run out. Queries queue. Page loads time out, the load balancer's health checks start failing on application servers that are stuck waiting on the database, and the cache stays empty because the requests that would fill it never finish.

That scenario is a composite, not one client's outage. Nothing in it is a bug in the usual sense. Every component did exactly what it was built to do. The design assumed the cache would be warm.

The Cache Became a Load-Bearing Wall

Cache-aside is the most common caching pattern on AWS, and the simplest. The application reads the cache. On a hit, it is done. On a miss, it reads the database, writes the result into the cache with a time to live (TTL), and returns it. The ElastiCache documentation calls it lazy loading.

The trouble starts with success. As the hit ratio climbs, the database looks oversized, so someone sizes it down. Now the cache is not an optimization. It is a load-bearing wall, and the system has two very different operating modes: warm, where everything is fast and cheap, and cold, where the database receives traffic it was never sized for.

Three things push a system into the cold mode:

• A hot key expires. Thousands of concurrent requests miss the same key in the same second, and every one of them runs the same query for the same row. This is the thundering herd, and it can happen on a perfectly healthy cluster.
• The whole cache goes cold. A node replacement without a replica, a failover to a new cluster, a flush, or a deploy that changes how keys are built.
• TTLs line up. Items loaded together, by a warm-up job or after a restart, all expire together, so the stampede repeats on a schedule.

Facebook's outage of September 23, 2010 is the public example of a cache feedback loop. An automated system checked cached configuration values and replaced invalid ones from the database. When the database copy was itself bad, every client that got an error treated the value as invalid and deleted the cache key, which sent still more queries to databases that could not recover. Facebook stopped all traffic to the database cluster, which meant turning off the site, to break the loop.

The Amazon Builders' Library article Caching challenges and strategies covers the same ground from inside Amazon: cold starts, thundering herds, and request coalescing. The lesson is the same at any scale. Once a service depends on its cache, losing the cache is a failure mode you have to design and test for.

Coalesce, Jitter, and Serve Stale

The fix is not a bigger cache. It is a cache-aside implementation that behaves well at the moment the cache is empty. Five techniques do most of the work.

Coalesce misses for the same key. When many callers miss the same key at once, exactly one should go to the database. Inside one process, this is single-flight: the first caller starts the load and the others wait on its result. Across a fleet, it is a short lease lock in the cache itself. The caller that wins SET key:lock token NX PX 5000 loads the row and fills the cache. The callers that lose either serve a stale copy or wait a few milliseconds and read the cache again. The lease expires on its own, so a loader that crashes cannot block the key forever, and the winner releases the lock only if the stored token is still its own.

Add jitter to every TTL. Instead of a fixed 300 seconds, use 300 to 360 seconds at random. Expirations spread out, and a warm-up job no longer schedules its own stampede.

Serve stale while one caller refreshes. Store a soft expiry inside the cached value and set a longer hard TTL on the key. After the soft expiry, the first caller to notice takes the lease and refreshes; everyone else keeps getting the slightly old value. For most product, profile, and catalog data, ten seconds of staleness is invisible to users. A database outage is not.

Cache negative results briefly. A lookup for a key that does not exist misses every time, so an attacker or a bad client can walk straight through the cache. Cache "not found" for 30 to 60 seconds.

Warm before you shift traffic. A new cluster, a new key format, or a new Region should be filled before it takes production reads, by replaying recent keys or by dual-writing for a while, and then ramped up.

None of this removes the need for the database to survive some cold-cache load. Coalescing turns ten thousand queries for one key into one query, but a cold cache with a million distinct hot keys is still a million queries. Cap the database's connections, and shed what it cannot serve (post 4) instead of letting it fall over.

Building It on AWS

On AWS, the lease lock lives in ElastiCache for Valkey or Redis OSS, which gives you atomic SET NX with an expiry in one command. Run the cluster with replicas in more than one Availability Zone, so a primary failure promotes a replica that already holds the data instead of starting cold. Put request collapsing at the edge with Amazon CloudFront, which already does it for you.

Here is the core of a coalescing read in Python with redis-py. The loser of the lock waits briefly and reads again; after three tries, it gives up and lets the caller shed the request rather than join a stampede.

import random, time
from uuid import uuid4

def get_product(r, db, pid, ttl=300):
    key, lock = f"product:{pid}", f"product:{pid}:lock"
    for _ in range(3):
        if (hit := r.get(key)) is not None:
            return decode(hit)
        token = uuid4().hex
        if r.set(lock, token, nx=True, px=5000):     # one loader per key
            try:
                row = db.load_product(pid)
                r.set(key, encode(row), ex=int(ttl * random.uniform(1.0, 1.2)))
                return row
            finally:
                release(r, lock, token)              # Lua: DEL only if value == token
        time.sleep(0.05)                             # someone else is loading
    raise CacheBusy(pid)                             # shed, don't stampede

When Not to Use It

Data that must be current for every read. An account balance, or inventory at the moment of checkout, should not come from cache-aside. Read it from the database of record, and cache only the views around it that can tolerate a few seconds of age.

Low-traffic keys. Coalescing costs a lock round trip on every miss. For keys that see a request a minute, plain cache-aside with jittered TTLs is enough.

Small data that rarely changes. A list of countries, a pricing table, or a feature catalog that changes weekly does not need a distributed cache. Load it into memory at startup, or publish it as a static file through CloudFront. If many query shapes need precomputed answers, a read model (post 7) is the better tool.

How We Check for It

This sits in the Performance Efficiency pillar, under PERF03, "How do you store, manage, and access data in your workload?", and its best practice PERF03-BP05, "Implement data access patterns that utilize caching." It also sits in the Reliability pillar under REL05-BP01, "Implement graceful degradation to transform applicable hard dependencies into soft dependencies," because a cache the database cannot run without is a hard dependency.

In a Vantalect review, we look at:

• What happened to database load the last time the cache was flushed or failed over in a test environment, and whether anyone has tried.
• TTL settings in code, and whether they carry jitter.
• CacheHitRate and Evictions in CloudWatch over the last month, and the database's headroom if the hit ratio fell to zero.
• Whether the cache runs with replicas across Availability Zones, and whether a failover has been tested.
• Whether hot keys are coalesced, or whether one popular item can send thousands of identical queries to the database.

If you want those answers for your own workloads, that is what the Well-Architected Review is for.

Related Patterns

• CQRS With Read Models: when caching one query shape is not enough, build a store shaped for the reads.
• Load Shedding and Backpressure: what the database should do when even coalesced misses are too many.
• Static Stability: a cache that keeps serving last-known-good data is a static-stability tool.
• Observability That Answers Questions: hit ratio and evictions belong on the same dashboard as database load.

This post is part 5 of Cloud Design Patterns.