Retry and Idempotency for FastAPI Background Tasks

Key takeaways:

  • Every queue worth using delivers at least once, so your job will eventually run twice — design for it.
  • An idempotency key derived from the business event turns a duplicate delivery into a cheap no-op.
  • Retry only non-deterministic failures: timeouts, connection resets, 429 and 5xx. Retrying a 400 is a waste.
  • Exponential backoff with jitter prevents a recovering dependency from being knocked over by a synchronised retry storm.
  • Give up eventually and dead-letter the job. A retry loop with no exit is an outage that hides itself.

A payment capture job failed once with a gateway timeout, retried, and charged the customer twice. This page — part of Background Task Processing — is about the two mechanisms that make that impossible: a retry policy that knows what is worth retrying, and an idempotency key that makes a second execution harmless.

Retry with backoff guarded by an idempotency key A job flows into an idempotency check, then through three attempts with growing backoff delays, ending either in a recorded result or a dead-letter queue. job delivered key=idem-1 seen this key? yes: return, do nothing attempt 1: fail attempt 2: fail attempt 3: ok wait 0.5s wait 1.0s store result under key budget spent: dead-letter
The idempotency check runs before any work; the retry loop runs inside it, so duplicates never reach the side effect at all.

Why At-Least-Once Delivery Forces Your Hand

Message brokers make a delivery guarantee, and the useful ones guarantee at least once. The reason is unavoidable: a worker must acknowledge a message after it finishes the work, because acknowledging first would lose the job if the worker crashed mid-execution. But acknowledging last means there is a window — between the side effect landing and the ack being written — in which a crash, a network partition, or a visibility-timeout expiry causes the broker to hand the same message to another worker.

That window is small and it is never zero. Celery's acks_late, SQS's visibility timeout, and ARQ's job expiry all have the same shape. Exactly-once delivery is not on offer; exactly-once effect is, and you build it yourself with an idempotency key.

The second source of duplicates is your own retry logic. If a gateway call times out, you do not know whether the request was processed. The timeout tells you that you did not get an answer, not that nothing happened. Retrying is correct, and it is also exactly how you double-charge someone — unless the downstream operation is keyed.

So the two mechanisms are complementary, and the order matters. Idempotency wraps retry, not the other way round: check the key once, then retry the work inside that guard. If you check the key inside the retry loop you re-read it on every attempt for no benefit, and if you store the result before the work completes you turn a transient failure into permanent silent data loss.

A Verified Retry Helper

Retry code is easy to write and hard to convince yourself is correct, because the interesting paths only run when something is broken. The example below makes the failure sequence deterministic — order-A fails twice then succeeds, order-B always fails — so the attempt counts and backoff delays can be observed rather than assumed.

import asyncio

from fastapi import BackgroundTasks, FastAPI

app = FastAPI()

ATTEMPTS: list[str] = []          # Every attempt, including the ones that fail.
CHARGES: list[str] = []           # Side effects that actually landed.
SLEEPS: list[float] = []          # Backoff delays, recorded instead of really slept.
PROCESSED: dict[str, dict] = {}   # The idempotency store, keyed by idempotency key.


async def sleep(delay: float) -> None:
    """Record the backoff instead of waiting, so the transcript is deterministic and fast."""
    SLEEPS.append(delay)
    await asyncio.sleep(0)


async def retry(fn, *args, attempts: int = 5, base: float = 0.5, cap: float = 8.0):
    """Call fn with exponential backoff. Raises the last error if every attempt fails."""
    last: Exception | None = None
    for attempt in range(1, attempts + 1):
        try:
            return await fn(*args, attempt=attempt)
        except Exception as exc:
            last = exc
            if attempt == attempts:
                break
            await sleep(min(cap, base * 2 ** (attempt - 1)))
    raise last

The idempotency guard sits outside the retry loop, and the task wrapper makes sure an exhausted budget lands somewhere a human will see it:

FAILURES_BEFORE_SUCCESS = {"order-A": 2, "order-B": 99}


async def charge_card(order_id: str, *, attempt: int) -> dict[str, object]:
    ATTEMPTS.append(f"{order_id} attempt {attempt}")
    if attempt <= FAILURES_BEFORE_SUCCESS.get(order_id, 0):
        raise ConnectionError(f"gateway timeout on {order_id} (attempt {attempt})")
    CHARGES.append(order_id)
    return {"order_id": order_id, "attempts_used": attempt}


async def charge_once(order_id: str, key: str) -> None:
    """At-least-once delivery means this may be invoked twice; the key makes that harmless."""
    if key in PROCESSED:
        ATTEMPTS.append(f"{order_id} skipped: idempotency key {key} already processed")
        return
    result = await retry(charge_card, order_id)
    PROCESSED[key] = result


DEAD_LETTER: list[str] = []


async def charge_task(order_id: str, key: str) -> None:
    """Never let a task raise into the server: exhausted retries go to a dead-letter list."""
    try:
        await charge_once(order_id, key)
    except Exception as exc:
        DEAD_LETTER.append(f"{order_id} key={key} gave up: {type(exc).__name__}: {exc}")


@app.post("/charge/{order_id}")
async def charge(order_id: str, key: str, tasks: BackgroundTasks) -> dict[str, str]:
    tasks.add_task(charge_task, order_id, key)
    return {"order_id": order_id, "idempotency_key": key, "status": "queued"}

Running that app with a job that recovers, a redelivery of the same job, and a job that never recovers produces this real output:

$ POST /charge/order-A?key=idem-1
200 OK
{
  "order_id": "order-A",
  "idempotency_key": "idem-1",
  "status": "queued"
}

$ POST /charge/order-A?key=idem-1
200 OK
{
  "order_id": "order-A",
  "idempotency_key": "idem-1",
  "status": "queued"
}

$ POST /charge/order-B?key=idem-2
200 OK
{
  "order_id": "order-B",
  "idempotency_key": "idem-2",
  "status": "queued"
}

$ GET /state
200 OK
{
  "attempts": [
    "order-A attempt 1",
    "order-A attempt 2",
    "order-A attempt 3",
    "order-A skipped: idempotency key idem-1 already processed",
    "order-B attempt 1",
    "order-B attempt 2",
    "order-B attempt 3",
    "order-B attempt 4",
    "order-B attempt 5"
  ],
  "charges": [
    "order-A"
  ],
  "backoff_delays": [
    0.5,
    1.0,
    0.5,
    1.0,
    2.0,
    4.0
  ],
  "idempotency_store": {
    "idem-1": {
      "order_id": "order-A",
      "attempts_used": 3
    }
  },
  "dead_letter": [
    "order-B key=idem-2 gave up: ConnectionError: gateway timeout on order-B (attempt 5)"
  ]
}

Three things in that transcript are worth naming. charges contains order-A exactly once despite four deliveries of work for it — that is the idempotency key doing its job. The backoff_delays list shows the doubling sequence twice: 0.5, 1.0 for the order that recovered on its third attempt, then 0.5, 1.0, 2.0, 4.0 for the one that burned all five attempts. And order-B produced a dead-letter entry instead of an exception escaping into the server, which is the behaviour described in when BackgroundTasks silently fails.

Making the Policy Production-Grade

The helper above is deliberately minimal. Three changes turn it into something you would run.

Add jitter. Pure exponential backoff synchronises clients: if a dependency returns errors for two seconds, every caller that failed at the same moment retries at the same moment. Full jitter — sleeping a random amount between zero and the computed delay — spreads the load out and is nearly always better than the deterministic schedule:

import random

delay = min(cap, base * 2 ** (attempt - 1))
await asyncio.sleep(random.uniform(0, delay))    # Full jitter.

Retry only what can succeed on a second try. A blanket except Exception retries validation errors five times before failing exactly as it would have failed immediately.

RETRYABLE = (ConnectionError, TimeoutError, asyncio.TimeoutError)


def is_retryable(exc: Exception) -> bool:
    if isinstance(exc, RETRYABLE):
        return True
    status = getattr(getattr(exc, "response", None), "status_code", None)
    return status in {429, 500, 502, 503, 504}   # Transient by definition.

Honour Retry-After when the upstream sends it — a 429 that tells you when to come back is more informative than your own backoff curve, and the semantics of that header are covered in rate limit headers and 429 responses.

Make the idempotency store durable and racy-safe. A dict is fine for a demonstration and useless across processes. In production the store is a table with a unique constraint or a Redis key set with NX:

async def claim(key: str, ttl_seconds: int = 86_400) -> bool:
    """Atomically claim a key. Returns False if another worker already has it."""
    return bool(await redis.set(f"idem:{key}", "in-progress", nx=True, ex=ttl_seconds))

SET NX is the important detail. Two workers can be handed the same message simultaneously, and a read-then-write check (if key in store) has a window between the read and the write where both see "not processed". The atomic claim closes it. When the work finishes, overwrite the value with the result so a later redelivery can return the original response instead of doing nothing at all — this is exactly how payment APIs implement client-supplied Idempotency-Key headers.

Where the side effect is a database write, you can often skip the separate store entirely: an INSERT ... ON CONFLICT DO NOTHING keyed by the business id is the idempotency check, and it is atomic for free.

Verification

Test the retry policy and the idempotency guard separately — they fail in different ways.

@pytest.mark.asyncio
async def test_retries_then_succeeds():
    calls = []

    async def flaky(*, attempt):
        calls.append(attempt)
        if attempt < 3:
            raise ConnectionError("boom")
        return "ok"

    assert await retry(flaky, attempts=5) == "ok"
    assert calls == [1, 2, 3]            # Stopped as soon as it worked.


@pytest.mark.asyncio
async def test_duplicate_delivery_is_a_no_op():
    await charge_once("order-1", key="k")
    await charge_once("order-1", key="k")
    assert CHARGES == ["order-1"]        # The side effect happened once.

In production, the signal is a pair of counters: task_attempts_total and task_dead_lettered_total. A healthy service has attempts slightly above job count and a dead-letter rate near zero. A rising ratio of attempts to jobs means a dependency is degrading well before it starts returning outright errors — one of the more useful early warnings you can have.

Trade-offs and When Not To

Retries convert a fast failure into a slow one. A job that retries five times with backoff occupies a worker slot for seconds; multiply by a queue full of jobs hitting the same broken dependency and your worker pool is saturated retrying work that cannot succeed. A circuit breaker in front of the retry loop is the standard answer: after N consecutive failures against a dependency, fail immediately for a cooling-off period instead of queueing more attempts.

Idempotency keys cost storage and a lookup on the hot path, and they need a TTL policy. Too short and a delayed redelivery slips past the guard; too long and the store grows without bound. A day is a common compromise for jobs whose redelivery window is minutes.

Finally, none of this belongs in BackgroundTasks for work that matters. Retry logic in-process is lost on restart along with the job itself, so a retry loop there gives you the illusion of durability without the substance. Choose the runner first — see FastAPI BackgroundTasks vs Celery vs ARQ and running ARQ workers with FastAPI — then let the broker own the retry schedule and use the code above only for the in-job retries the broker cannot see.

FAQ

Why is idempotency mandatory rather than nice to have? Because every practical queue delivers at least once. A worker can complete the side effect and then die before acknowledging the message, so the broker redelivers work that already succeeded. Only an idempotent job survives that without double-charging or double-sending.

What should I use as an idempotency key? A value derived from the business event, not from the attempt: an order id, a payment intent id, or a hash of the request body plus the client-supplied Idempotency-Key header. It must be identical across every redelivery of the same logical job.

How many retries and how much backoff? Enough retries to ride out a short dependency blip, and no more. A common shape is five attempts with exponential backoff from about half a second, capped at a few seconds, with jitter. After that, dead-letter the job so a human can see it.

Which errors should not be retried? Anything deterministic. A 400, a validation failure, or a missing row will fail identically on every attempt and just burns your retry budget. Retry timeouts, connection errors, 429s and 5xx responses; fail fast on everything else.

Does FastAPI BackgroundTasks retry failed jobs? No. It has no retry, no persistence and no dead-letter queue. Any retry logic you add there runs inside the same process and is lost if that process restarts, which is why durable work belongs on a broker-backed queue.