Background Task Processing in FastAPI

Background task processing is moving work the client does not need for its response out of the request path — using FastAPI's BackgroundTasks for short in-process side effects, and a broker-backed queue such as Celery or arq for anything that must not be lost.

This topic is part of Async, Background Tasks and Observability. It is the pressure-release valve for async correctness: once you accept that slow work on the loop harms every concurrent request, the next question is where that work should go instead.

The decision is genuinely binary, and people get it wrong in both directions. Reaching for Celery to send a Slack message adds a broker, a worker deployment and a monitoring surface to a problem that needed four lines. Using BackgroundTasks to capture a payment loses money on the next deploy. The rest of this guide is about knowing which side of that line you are on, starting with what BackgroundTasks actually is — because it is smaller than most people think.

Prerequisites

• FastAPI 0.139.2 on Python 3.12.
• For durable queues, a broker you are willing to operate — Redis for arq, Redis or RabbitMQ for Celery.
• An understanding of the worker's concurrency model from Async Correctness and Concurrency, because background tasks share it.

Transcripts below come from running BackgroundTasks in-process through the verification harness. Broker-backed behaviour is described rather than executed, since no broker is available in that environment.

Core Mechanics: BackgroundTasks Is a List and a For Loop

BackgroundTasks is far less machinery than the name suggests. It collects callables, and Starlette awaits them one at a time after the response has been written. There is no scheduler, no concurrency, no persistence, and no supervision.

from fastapi import BackgroundTasks


@app.post("/comments")
async def add_comment(body: CommentIn, tasks: BackgroundTasks) -> dict[str, str]:
    comment = await save_comment(body)
    # Runs after the response. If the process dies first, the notification is simply lost.
    tasks.add_task(notify_subscribers, comment.id)
    return {"id": comment.id}

Three properties follow from that implementation, and each one surprises somebody.

They run in order, one at a time

Three tasks that each sleep 10ms:

$ GET /probe/task-ordering
200 OK
{
  "each_task_sleeps_s": 0.01,
  "completion_order": [
    "task 1 finished",
    "task 2 finished",
    "task 3 finished"
  ],
  "asgi_call_took_s": 0.05,
  "would_be_s_if_concurrent": 0.01,
  "would_be_s_if_sequential": 0.03
}

0.03 seconds, not 0.01. They are awaited sequentially in registration order, not gathered. Ordering guarantees are occasionally useful, but the practical consequence is that one slow task delays every task behind it, and one failing task means the rest never run at all.

Sync and async task functions dispatch differently

The same rule that governs endpoints governs tasks:

$ GET /probe/where-tasks-run
200 OK
{
  "handler_thread": "MainThread",
  "tasks_ran_on": [
    "sync-task on AnyIO worker thread",
    "async-task on MainThread"
  ],
  "threadpool_capacity": 40
}

A plain def task ran on AnyIO worker thread; an async def task ran on MainThread, the loop thread. This is good news and a trap in equal measure. Good, because a synchronous task function does not block the loop — you can hand BackgroundTasks a legacy sync function safely. A trap, because an async def task containing a blocking call does block the loop, exactly as an endpoint would, and it does so after the response has gone, where it is much harder to attribute.

They cost the worker, after the client has gone

$ GET /probe/capacity-ceiling
200 OK
{
  "task_duration_s": 0.1,
  "one_request_holds_the_worker_for_s": 0.1,
  "ten_concurrent_requests_hold_it_for_s": 0.1,
  "tasks_completed": 11
}

The ASGI call does not complete until the tasks have run: a request whose task takes 100ms occupies the worker for at least 100ms, even though the client already has its response and has moved on.

This is the single most under-appreciated cost of BackgroundTasks. The work is invisible in your request-duration metrics — those stop when the response is sent — but perfectly visible in your capacity. The symptom is elevated latency on other endpoints with nothing in the traces to explain it. The healthier ten-concurrent number reflects the fact that awaited tasks from different requests do overlap; the sequential constraint applies within a request, not across them.

Production Implementation: Durable Queues

When the work must not be lost, the request should do nothing but record the intent durably and return. A separate worker pool does the actual work, with retries.

# arq worker: async-native, which suits an async-first FastAPI codebase.
from arq.connections import RedisSettings


async def generate_report(ctx: dict, report_id: str) -> None:
    """Idempotent by construction: safe to run twice because it upserts by report_id."""
    await build_and_store_report(report_id)


class WorkerSettings:
    functions = [generate_report]
    redis_settings = RedisSettings()
    max_tries = 5             # the broker owns the retry schedule

@app.post("/reports")
async def create_report(request: Request) -> dict[str, str]:
    report_id = new_id()
    await request.app.state.arq_pool.enqueue_job(
        "generate_report", report_id, _job_id=report_id,   # job id doubles as a dedupe key
    )
    return {"report_id": report_id, "status": "queued"}

Two design points matter more than the choice of library.

Enqueue identifiers, not objects. Pass a report_id, not a loaded ORM instance and never a database session. The worker re-loads what it needs in its own transaction. Handing a request-scoped session to background work is a specific and expensive mistake — see Async SQLAlchemy Session per Request.

Assume at-least-once delivery. Every broker can deliver a job twice, because the window between doing the work and acknowledging it is never zero. That is not a defect to configure away; it is a property to design for, by making jobs idempotent. Retry and Idempotency for Tasks covers the idempotency key, backoff and dead-letter handling.

The full comparison of BackgroundTasks, Celery and arq — including deployment shape and operational cost — is in FastAPI BackgroundTasks vs Celery vs arq, and the worker-and-app wiring in Running arq Workers with FastAPI.

Async and Performance Notes

Deferring is not offloading. BackgroundTasks moves work later in the same process; it does not move it elsewhere. Your worker still pays for it, and the deferral only helps if the client's latency was the problem.

A CPU-bound background task is worse than a CPU-bound handler. Same cost to the worker, but now invisible to your request metrics. CPU work belongs in a process pool or a queue.

Queue depth is a leading indicator. Request latency tells you nothing about a broker-backed system's health; the queue absorbs the problem until it cannot. Alert on depth and on oldest-job age, not just on error rate.

Do not let workers share the web tier's connection budget silently. Every worker process opens its own database pool, and those connections count against the same max_connections. See Fixing asyncpg Connection Pool Exhaustion.

Testing Strategy

BackgroundTasks is unusually pleasant to test, because of the ordering property above: since the ASGI call does not return until the tasks have run, the side effects have already happened by the time you hold the response object. No sleeping, no polling.

def test_signup_writes_the_audit_line(client):
    AUDIT.clear()
    response = client.post("/signup", json={"email": "ada@example.com"})
    assert response.status_code == 200
    assert AUDIT == ["signup:ada@example.com"]     # already done, no waiting

Assert on the observable effect — a row written, a message enqueued, a counter moved — rather than on a mock having been called. The mock assertion keeps passing after a refactor that breaks the behaviour.

One asymmetry to know about: TestClient re-raises exceptions from background tasks by default, while a real server does not. A task that fails silently in production will fail loudly in CI. That is a useful detector rather than a nuisance, and the details are in When BackgroundTasks Silently Fails.

For broker-backed jobs, split the test in two. Test that the endpoint enqueues the right job with the right arguments, using a fake pool. Then test the job function directly as a plain async function, including calling it twice to prove idempotency. Testing the broker itself is the broker maintainers' job.

Failure Modes and Diagnosis

Work vanishes after a deploy. BackgroundTasks holding something important. Rolling restarts drop in-flight tasks silently, and the loss correlates with deploys rather than with load, which makes it easy to misattribute.

A task raises and nobody notices. The response was already sent, so the client sees a 200. The exception surfaces on the ASGI server's logger, not your application's, and every task behind it is skipped.

Unrelated endpoints get slower after adding a task. The worker is doing task work that no request metric attributes to anything.

A retried job double-charges. At-least-once delivery meeting a non-idempotent job. Key the effect, do not just hope the retry does not happen.

Jobs succeed but nothing changes. Frequently a worker running old code, or one connected to a different broker database than the producer. Log the job's code version on both sides.

Correlation IDs disappear in the worker. The contextvar set by tracing middleware does not cross the boundary. Pass it explicitly, per Correlating Logs, Traces and Errors.

The queue grows without bound. Consumers are slower than producers, or dead. Depth alone is ambiguous; oldest-job age tells you which.

Choosing Where the Work Goes

The last row is worth stating explicitly, because it is a common miscategorisation. If the client needs the result, it is not background work; it is concurrent work within the request.

FAQ

When should I use BackgroundTasks versus Celery or arq? Use BackgroundTasks for short, non-critical work whose loss on a restart is acceptable, such as a notification or an audit line. Use a durable queue when the work must survive a crash, be retried, run on a schedule, or scale independently. The dividing line is durability, not duration.

Do background tasks run in parallel with each other? No. BackgroundTasks awaits each task in turn in a plain loop. Three tasks sleeping 10ms each took 0.03 seconds rather than 0.01, and they completed strictly in registration order. One slow task delays every task queued behind it.

Does a sync task function block the event loop? No. A plain def task is dispatched to a worker thread just as a def endpoint is, while an async def task is awaited on the loop. Measured directly, the sync task ran on a thread named AnyIO worker thread and the async task ran on MainThread.

Do background tasks cost request capacity? Yes. The ASGI call does not complete until the tasks have run, so a request queuing a 100ms task occupies the worker for at least 100ms after the client has its response. The latency lands on other requests, with nothing in request metrics to explain it.

What happens if a background task raises? The client still gets its successful response, because the response was already committed before the task ran. The exception escapes into the ASGI server's logger rather than your application's, and every task queued behind the failing one is skipped.

How do I keep the correlation ID in a background job? Capture it when you enqueue and pass it as an explicit argument, then bind it inside the worker. Background work runs outside the request context, so a contextvar set by tracing middleware is not reliably available once the request has finished.

Related

• Up to the area: Async, Background Tasks and Observability.
• The comparison: FastAPI BackgroundTasks vs Celery vs arq on durability, ecosystem and operational cost.
• The deployment: Running arq Workers with FastAPI wires a worker to the app.
• The silent failure: When BackgroundTasks Silently Fails on where the exception goes and how to see it.
• Surviving delivery: Retry and Idempotency for Tasks covers backoff, idempotency keys and dead letters.
• The prerequisite: Async Correctness and Concurrency, because tasks share the worker's loop.