Caching Strategies in FastAPI

Caching is the deliberate trade of freshness for speed: keeping hot, slow-changing data somewhere fast so most requests never reach the database, and accepting that some readers will see data that is slightly out of date.

This topic is part of Async, Background Tasks and Observability. It is the main relief valve for async database sessions under read-heavy load — every cache hit is a connection not checked out — and it composes with serialization performance when what you cache is the rendered response rather than the row.

The framing that makes caching tractable is that a cache has exactly two hard problems, and neither is the lookup. The first is who sees stale data and for how long, which is a product decision wearing an engineering costume. The second is what happens at the moment the cache does not help — the cold key, the expired key, the deploy that flushed everything. Almost every cache incident is the second problem.

Prerequisites

• FastAPI 0.139.2 on Python 3.12, with an async client for whatever store you choose.
• A read-heavy endpoint with a defensible staleness tolerance. Caching something nobody reads twice is pure overhead.
• Metrics you can attribute per key prefix, or you will not know whether the cache is working.

The transcripts below were produced in-process against an async dictionary cache with a TTL, so every count and timing is real. Redis is not available in this page's verification environment, so the Redis command mappings (SET, EX, NX, DEL) are described rather than executed; the semantics shown are the ones those commands implement.

Core Mechanics: Cache-Aside

The application owns the logic. Check the cache, fall through on a miss, populate, return. Nothing is implicit, which is what makes it debuggable.

async def get_product(product_id: int) -> dict:
    key = f"product:{product_id}"
    if (cached := cache_get(key)) is not None:
        STATS["hits"] += 1
        return cached                       # hit: the database is never touched
    STATS["misses"] += 1
    value = await load_from_db(product_id)
    cache_set(key, value, TTL_SECONDS)      # populate with an expiry, always
    return value

Four reads of the same key, through the endpoint, with the loader taking 100ms:

$ GET /probe/hit-and-miss
200 OK
{
  "loader_takes_ms": 100,
  "first_request_ms": 100,
  "next_three_requests_total_ms": 0,
  "database_loads": 1,
  "hits": 3,
  "misses": 1,
  "body": {
    "id": 7,
    "name": "product-7",
    "loaded_at_call": 1
  }
}

One database load, three hits. The first request paid the loader's full 100ms; the next three together did not register a single millisecond. That gap is why caching is worth the complexity, and it is also the number that makes people over-confident — the steady state is easy.

Against Redis the same shape uses GET, then SET key value EX 300. The only structural difference is that the value must be serialized and the calls must be awaited on an async client, because a synchronous Redis client blocks the loop on every request rather than only on misses.

The Stampede Is the Real Problem

Now take the same code and give it twenty concurrent readers on a key that is not yet populated:

$ GET /probe/stampede
200 OK
{
  "concurrent_readers": 20,
  "database_loads": 20,
  "hits": 0,
  "misses": 20,
  "elapsed_s": 0.1
}

Twenty loads. Zero hits. Every reader checked the cache, found nothing, and went to the database — because the first one had not finished loading, let alone populating, when the other nineteen looked.

This is the failure that takes services down, and note how invisible it is in ordinary conditions. The cache reports an excellent hit rate all day. Then a key expires during peak traffic, or a deploy flushes the cache, or a popular item trends, and your database receives your full unbuffered concurrency in one instant. The cache did not merely fail to help; it concentrated the load into a spike.

The fix is single-flight: one lock per key, and a second check inside it.

async def get_product_single_flight(product_id: int) -> dict:
    key = f"product:{product_id}"
    if (cached := cache_get(key)) is not None:
        STATS["hits"] += 1
        return cached
    lock = LOCKS.setdefault(key, asyncio.Lock())
    async with lock:
        # Re-check inside the lock: the winner may have populated the key while we waited.
        if (cached := cache_get(key)) is not None:
            STATS["hits"] += 1
            return cached
        STATS["misses"] += 1
        value = await load_from_db(product_id)
        cache_set(key, value, TTL_SECONDS)
        return value

The same twenty readers:

$ GET /probe/single-flight
200 OK
{
  "concurrent_readers": 20,
  "database_loads": 1,
  "hits": 19,
  "misses": 1,
  "elapsed_s": 0.1
}

Twenty loads became one, and nineteen readers got hits. The number worth dwelling on is elapsed_s: 0.1 — identical to the unprotected run. Nobody waited longer. Nineteen requests that would have each run their own 100ms query instead waited on one query that was already in flight.

The double-check inside the lock is not optional. Without it, every waiter re-runs the loader after acquiring the lock, and you have serialised the stampede rather than eliminated it.

Two scaling caveats. An asyncio.Lock coordinates one worker; four workers still produce four loads. For genuine cross-process single-flight, use a Redis SET lock:key value NX EX 5 as the lock, where NX makes acquisition atomic. And a per-key lock dictionary that is never pruned is a slow memory leak, so drop the lock once the key is populated.

TTL and Invalidation

A TTL is the staleness contract, and it should be chosen from the product requirement rather than from habit. Watch what it actually does — the key here has a 0.3s TTL:

$ GET /probe/ttl-expiry
200 OK
{
  "ttl_s": 0.3,
  "timeline": [
    "t=0.00s  loads=1 hits=0",
    "t=0.15s  loads=1 hits=1",
    "t=0.45s  loads=2 hits=1"
  ]
}

Inside the window, a hit. Past it, a second load. An expired key is indistinguishable from a cold one — which is precisely why expiry and stampedes are the same problem, and why hot keys deserve either a lock or proactive refresh before expiry.

TTL alone is a blunt instrument when you control the writes. Deleting the key on write bounds staleness by the write rather than by the clock:

@app.post("/products/{product_id}")
async def update_product(product_id: int, name: str) -> dict:
    """Write, then delete the key. The next reader repopulates from the source of truth."""
    await write_product_to_db(product_id, name)
    STORE.pop(f"product:{product_id}", None)          # Redis: DEL product:{id}
    return {"updated": product_id, "name": name}

$ GET /probe/invalidation
200 OK
{
  "second_read_served_from_cache": true,
  "cache_key_survived_the_write": false,
  "database_loads_total": 2,
  "hits": 1,
  "misses": 2
}

The read after the write missed and reloaded, so no client saw the pre-write value. The right posture is invalidate on write, and keep a TTL as the safety net for the invalidation paths you forgot — the admin tool, the migration, the other service writing to the same table. A key with no expiry at all serves stale data forever the first time an invalidation is missed, which is the most common serious caching bug and the easiest to prevent.

Where a value is expensive to invalidate precisely, version the key prefix instead: moving every reader from product:v1:{id} to product:v2:{id} invalidates everything atomically at deploy time, with no scan and no flush.

Choosing a Cache Scope

Cache-aside says how; it does not say where. Three scopes, in increasing order of reach and risk:

The request scope is free — FastAPI caches dependency results within a request, so a dependency declared in five places runs once. The process scope is right for things that are the same for everybody and wrong, sometimes dangerously, for anything keyed by user. The full treatment, including a measured demonstration of one user receiving another's permissions from a badly keyed process cache, is in Caching Dependency Results.

The rule that prevents the worst outcome: if the value depends on who is asking, the identity must be in the key, at every scope, without exception.

Async and Performance Notes

Use an async client. A synchronous cache call blocks the loop on every request, including hits, which is the majority. That converts your fastest path into your most frequent blocking call.

Cache the representation that removes the most work. A serialized response also skips re-serialization, which is significant for large nested payloads; a cached model is more reusable across endpoints. Profile before choosing.

Keep values small. Large values cost network time on every hit and can make a cache slower than the database for small reads. A primary-key lookup on a warm connection may already beat a Redis round trip.

Do not cache write paths or already-fast reads. Caching has a fixed cost per operation, and applying it to something that takes 2ms makes it slower.

Testing Strategy

Assert on loader call counts, not on timing. Counts are deterministic; timings flake.

async def test_second_read_does_not_touch_the_database(cache, db):
    await get_product(1)
    db.calls.clear()
    await get_product(1)
    assert db.calls == []                     # served from cache


async def test_write_invalidates(cache, db):
    await get_product(1)
    await update_product(1, {"name": "new"})
    assert cache_get("product:1") is None      # the next read repopulates

Three tests are worth writing that most suites do not have. A stampede test that fires N concurrent cold readers and asserts the loader ran once — the only way to catch a missing lock. A key-scoping test that reads as two different users and asserts they get different values, which is the test that catches a data leak. And an expiry test using an injectable clock rather than sleep, so it is fast and deterministic.

Fake the cache rather than mocking it. A dictionary with a TTL behaves like the real thing for every assertion above and needs no running server. Reserve a real Redis for one integration test that proves serialization round-trips.

Failure Modes and Diagnosis

A stampede on a hot key. Latency spike plus a database load spike at the moment a key expires. Add single-flight or refresh hot keys before expiry.

Keys with no TTL. One missed invalidation and the value is wrong forever. Set an expiry on every key, always.

User data cached under a shared key. Not a stale read — the wrong person's data, frequently used for an authorisation decision. Include identity in the key.

A hit rate near 100% on user-specific data. Usually a missing key segment rather than good luck. Investigate it as a bug.

Cache and database disagree after a deploy. Two versions of the code writing different value shapes to the same key. Version the key prefix as part of the deploy.

Thundering herd after a restart. Every worker starts cold at once. Stagger warming, or accept the spike knowingly.

A synchronous client blocking the loop. Uniform latency growth on every endpoint that touches the cache. See Async Correctness and Concurrency.

FAQ

What is the cache-aside pattern? The application checks the cache, and on a miss it loads from the source, stores the result and returns it. The cache sits beside the database rather than in front of it, so the application decides exactly what is cached and for how long. Measured over four reads of one key, it produced one database load and three hits.

What is a cache stampede and how bad is it? It is what happens when a popular key is cold or has just expired and many concurrent requests all miss at once. Measured with twenty concurrent readers on a cold key, every single one reached the database: twenty loads, zero hits. The cache provided no protection at the moment protection mattered most.

How do I stop a stampede? Put a per-key lock in front of the loader so only one caller computes while the others wait and then read the populated value. The same twenty concurrent readers went from twenty database loads to one, with nineteen hits, and the whole batch still finished in the time of a single load.

How do I choose a TTL? Set it to the longest staleness the use case genuinely tolerates, then treat that number as a contract. Short TTLs keep data fresh but lower the hit rate and reduce the relief the cache provides. For data that changes on a known event, invalidate explicitly instead of relying on expiry.

Should I invalidate on write or wait for the TTL? Invalidate on write whenever the write path is yours. Deleting the key makes the next read repopulate from the source, which bounds staleness by the write rather than by the clock. TTL then becomes a safety net for the invalidations you miss, not the primary mechanism.

Should I cache the database row or the serialized response? Cache whichever removes the most repeated work. A serialized response also skips re-serialization, which matters for large nested payloads but is harder to reuse across endpoints. A cached row or model is more reusable. Profile where the time actually goes before choosing.

Related

• Up to the area: Async, Background Tasks and Observability.
• The Redis build-out: Redis Response Caching in FastAPI takes cache-aside to a real store.
• Keeping it correct: Cache Invalidation Patterns in FastAPI on write-through, versioning and event-driven eviction.
• Choosing a scope: Caching Dependency Results measures request, process and Redis scopes, and the leak the middle one causes.
• What caching relieves: Async Database Sessions, since every hit is a connection not checked out.
• Where the CPU goes: Performance Optimization for Models if serialization rather than the query is your cost.