Background Task Processing in FastAPI

Background task processing is moving work the client does not need for its response out of the request — using FastAPI's BackgroundTasks for short fire-and-forget jobs and a real queue such as Celery or ARQ for durable, retryable work.

This topic is part of Async, Background Tasks and Observability. It is the relief valve for async correctness: work that is slow or blocking is best removed from the request entirely, and durable jobs keep the hot path fast.

BackgroundTasks runs in-process with no durability; a broker-backed queue survives restarts and retries failed jobs on dedicated workers.

Core Mechanics: BackgroundTasks

BackgroundTasks schedules a callable to run after the response is sent, in the same process. It is ideal for short, non-critical side effects where losing the occasional job on a restart is acceptable.

from fastapi import BackgroundTasks


@app.post("/comments")
async def add_comment(body: CommentIn, tasks: BackgroundTasks) -> dict[str, str]:
    comment = await save_comment(body)
    # Notify after responding; if the process dies, a missed notification is fine.
    tasks.add_task(notify_subscribers, comment.id)
    return {"id": comment.id}

Production Implementation: Durable Queues

When work must not be lost — a payment capture, a report that takes minutes, anything that needs retries — use a broker-backed queue. The request enqueues a job and returns; a separate worker pool consumes it with retry semantics.

# ARQ worker: an async-native queue that fits FastAPI's async model.
from arq import create_pool
from arq.connections import RedisSettings


async def generate_report(ctx: dict, report_id: str) -> None:
    # Idempotent: safe to retry because it upserts by report_id.
    await build_and_store_report(report_id)


class WorkerSettings:
    functions = [generate_report]
    redis_settings = RedisSettings()
    max_tries = 5            # The broker retries on failure with backoff.

# Enqueue from a route, forwarding the correlation ID for traceability.
@app.post("/reports")
async def create_report(req: Request) -> dict[str, str]:
    pool = req.app.state.arq_pool
    report_id = new_id()
    await pool.enqueue_job("generate_report", report_id,
                           _job_id=report_id)   # Dedupe key prevents duplicates.
    return {"report_id": report_id, "status": "queued"}

The end-to-end comparison and worker deployment are detailed in FastAPI BackgroundTasks vs Celery vs ARQ.

Async and Performance Notes

BackgroundTasks runs on the same event loop after the response, so a slow task still consumes that worker's capacity — it defers work but does not move it off the machine. A real queue moves work to separate workers, which is what lets you scale CPU-heavy jobs independently of request traffic. Choose ARQ when your codebase is async-first and Celery when you need its mature ecosystem and scheduling.

Testing Strategy

Assert the response does not wait on the deferred work, and that jobs are idempotent:

def test_signup_responds_before_email(client, fake_mailer):
    resp = client.post("/signup", json={"email": "ada@example.com"})
    assert resp.status_code == 200          # Returns without awaiting delivery.
    # The mailer is invoked by the background task, not during the request.

Failure Modes and Debugging

Lost work on restart. Using BackgroundTasks for critical jobs drops them on deploy; move to a durable queue.
Non-idempotent retries. A retried job that is not idempotent double-charges or double-sends; key it.
Lost context. The correlation ID is gone in the worker unless forwarded; pass it as an argument, per Observability and Tracing.
Unbounded queues. A queue with no limits hides a failing worker; monitor depth and set alerts.

Up to the section: Async, Background Tasks and Observability.
Hands-on guide: FastAPI BackgroundTasks vs Celery vs ARQ.
Composes with: Async Correctness and Concurrency and Observability and Tracing.