FastAPI BackgroundTasks vs Celery vs ARQ

Key takeaways:

• The decision is not about speed. It is about what survives the process being replaced, which is a question with a factual answer for each option.
• BackgroundTasks passes live Python objects; a queue passes bytes. That difference dictates how your task signatures must be written.
• Retries require somewhere durable to keep an attempt counter, which is why BackgroundTasks cannot have them at any level of effort.
• ARQ and Celery differ mainly in ecosystem breadth versus configuration surface, not in capability.
• Mixing is normal: optional work in-process, work that must not be lost in the queue.

This comparison operationalizes Background Task Processing. If you have already chosen ARQ, the implementation is in running ARQ workers with FastAPI.

What Was Executed on This Page

Celery, ARQ and Redis are not installed in this site's verification environment, and there is no broker to talk to. This page therefore publishes no benchmark, no throughput figure and no worker log — inventing them is exactly what the site exists not to do, and a latency number produced on someone else's hardware would not help you choose anyway.

What is executed is the boundary that actually distinguishes the three options. BackgroundTasks is a real FastAPI feature that runs fully in-process, so its behaviour is directly observable. The queue side is represented by a store holding the encoded payload that a broker would hold, which is enough to demonstrate the two consequences that matter: what a queue can accept as an argument, and what is still there after the process is replaced. Everything attributed to a real Celery or ARQ deployment — prefetch, acknowledgement modes, beat scheduling, result backends — is described in prose.

The Problem This Solves

"Do this after responding" reads like one requirement. It is at least four: run it off the request path, run it even if this process dies, run it again if it fails, and run it on a machine that is not serving traffic. BackgroundTasks provides the first. The other three are the entire reason brokers exist, and choosing wrongly costs you either dropped work or infrastructure you did not need.

The usual failure is to pick BackgroundTasks because it is one import, ship it, and discover the gap during an incident rather than during design.

Why It Happens: One Boundary, Three Consequences

BackgroundTasks is a list of callables held on the Response object, awaited by Starlette after the body has been flushed. There is no encoding step, no external store and no coordinator. That is its entire implementation, and it explains all three of its limitations at once.

No encoding means anything can be an argument. You can pass a SQLAlchemy row, an open httpx.AsyncClient, a closure over request state. Nothing objects, because the object never leaves the heap. That freedom is pleasant right up to the day you migrate, at which point every task signature has to be rewritten to take identifiers.

No external store means no durability. The queue is a Python list in one process's memory. A rolling deploy, an OOM kill, a SIGTERM with a short grace period — any of these discards it with no record that work was pending. The client already has its 200.

No coordinator means no retries. A retry needs somewhere to record that attempt two is happening, and somewhere for the job to sit between attempts. Both of those are the broker. You can wrap a task in a for loop, but that loop dies with the process too, so it buys you nothing against the failure mode that matters.

Celery and ARQ solve all three by moving the job outside the process before the response is sent. The cost is precisely the freedom you gave up in the first consequence, plus a broker to run, plus a worker tier to deploy and observe.

The Durability Boundary, Executed

The following is a real run of an app that exercises both paths in-process: BackgroundTasks for the in-memory route, and an encoded payload store standing in for a broker on the queued route.

Handing BackgroundTasks a live object graph works without complaint:

$ POST /signup/background?email=ada@example.com
200 OK
{
  "email": "ada@example.com",
  "runner": "BackgroundTasks",
  "argument_crossed_a_process_boundary": false
}

Handing a queue the identifier works too, and shows what the broker would actually be holding:

$ POST /signup/queued?email=ada@example.com
200 OK
{
  "email": "ada@example.com",
  "runner": "queue",
  "enqueued_bytes": "{\"function\": \"send_welcome\", \"args\": [\"ada@example.com\"]}"
}

Handing the queue the same live object that BackgroundTasks accepted does not:

$ POST /signup/queued-object?email=ada@example.com
422 Unprocessable Entity
{
  "detail": "TypeError: Object of type Customer is not JSON serializable"
}

This is the migration tax, made concrete. The Customer instance holds a lambda, which no codec will encode — and while ARQ's default pickle codec is more permissive than the JSON used here, it fails on open connections and sessions just the same. Every task you wrote against BackgroundTasks must be re-expressed in terms of identifiers before it can be queued, and the task body must re-load what it needs from the database on the worker side.

Now the part the comparison hinges on. Both mechanisms have done their work, and the state looks equivalent:

$ GET /evidence
200 OK
{
  "in_process_done": [
    "notified ada@example.com"
  ],
  "broker_contents": [
    "{\"function\": \"send_welcome\", \"args\": [\"ada@example.com\"]}"
  ]
}

Then the process is replaced, as it is on every deploy:

$ POST /simulate-restart
200 OK
{
  "in_process_state_discarded": [
    "notified ada@example.com"
  ],
  "broker_depth_after_restart": 1
}

$ GET /evidence
200 OK
{
  "in_process_done": [],
  "broker_contents": [
    "{\"function\": \"send_welcome\", \"args\": [\"ada@example.com\"]}"
  ]
}

The in-process side is empty. The broker side is untouched, and a worker starting afterwards still has everything it needs:

$ POST /broker/drain
200 OK
{
  "completed_after_restart": [
    "notified ada@example.com"
  ]
}

To be exact about what this does and does not prove: the restart is simulated by discarding in-process state while leaving the encoded payloads intact, which is a model of what a real deploy does to a real process, not a real deploy. The property being demonstrated is structural — one mechanism keeps work in a heap that dies with the process and the other keeps it in bytes that do not — and that property is a fact about where the data lives, independent of any broker's implementation.

The Decision Table

Read the rows as a filter rather than a scoreboard. "Survives a restart" eliminates one option outright for most work that matters. "Extra infrastructure" and "config surface" then decide between the remaining two, and those are questions about your team, not your code.

Choosing, In Practice

Use BackgroundTasks when the honest answer to "what if this never runs?" is "nothing". Cache warming, a best-effort webhook, an analytics event, invalidating a key you will re-derive anyway. Keep the work sub-second, because it occupies a worker's concurrency slot after the client has gone, and guard the callable so a failure is logged rather than silently swallowed — the mechanics of that are in when BackgroundTasks silently fails.

Use ARQ when the codebase is async, the broker is Redis, and the task surface is small enough to hold in your head. It is async-native, so tasks are ordinary coroutines that can use the same async database and HTTP libraries as your routes, and cron lives in the worker instead of a separate process. The implementation guide is running ARQ workers with FastAPI.

Use Celery when you need what its ecosystem provides. A broker that is not Redis; routing rules across multiple queues with different priorities; canvas primitives for chaining and grouping work; the operational tooling and the very large corpus of answers to production questions. If you are running Celery already and it is not hurting, there is no argument here for migrating.

Consider the option none of these three represent. For work that is genuinely part of a transaction — the outbox pattern, where the job row is written in the same commit as the business change — a database table polled by a worker beats all three, because it eliminates the window where the write succeeded and the enqueue did not. That window is real for both ARQ and Celery, and it is the reason a payment confirmation should not be enqueued from a route that has not committed yet.

Verification

The in-process option has one test worth writing, which documents that the response does not wait:

def test_response_does_not_wait_for_the_task(client):
    resp = client.post("/audit", json={"type": "login"})
    assert resp.status_code == 200

For either queue, test the enqueue call and the task body separately. The enqueue test asserts that a route put the right job on the queue, using a fake pool through dependency_overrides so no broker is needed — the technique is covered in overriding dependencies in tests and mocking external services in tests:

async def test_route_enqueues_the_right_job(client, fake_queue):
    await client.post("/invoices/inv-1/sync")
    assert fake_queue.jobs == [("sync_invoice", "inv-1")]


async def test_arguments_survive_a_round_trip(fake_queue):
    # The check that catches a migration from BackgroundTasks before production does.
    encoded = fake_queue.encode("sync_invoice", "inv-1")
    assert fake_queue.decode(encoded) == ("sync_invoice", "inv-1")

The task body is then an ordinary async function test. Running it twice with the same input and asserting the result is unchanged is the test that makes at-least-once delivery safe, and it is worth writing before you enable retries rather than after.

Trade-offs and When Not To Use a Queue

A queue moves failure rather than removing it. Jobs that fail every attempt land in a dead-letter destination that somebody has to look at. If nobody owns that queue, a broker gives you the same silent loss as BackgroundTasks, only with more infrastructure and a longer delay before anyone notices.

Two deployables must stay in step. The worker resolves task names against its own registry, so an API that deploys ahead of the worker enqueues names the worker does not know. Deploy the worker first when adding a task, and the API first when removing one.

The connection budget doubles. Worker processes build their own database pools. A queue that scales out under load will happily exhaust the database while the API is still healthy — see fixing asyncpg pool exhaustion.

At-least-once means every job can run twice. This is not an edge case; it is the normal consequence of a timeout followed by a retry, and it applies equally to ARQ and Celery. Design for it with a natural key per job, as covered in retry and idempotency for tasks.

Do not adopt a broker for one task. If exactly one endpoint has deferred work and losing it is merely annoying, the operational cost of a worker tier exceeds the benefit. Revisit when the second task appears, which is usually when the requirement becomes real.

FAQ

Can BackgroundTasks replace Celery for production work? Only for work that is acceptable to lose. BackgroundTasks holds the callable in the memory of the process that served the request, with no persistence, no attempt counter and no separate scaling. The moment a job must survive a restart, retry on failure, or run on a schedule, you need a broker-backed queue.

Should I pick Celery or ARQ for an async FastAPI app? ARQ is async-native, Redis-only and small enough to read in full, which suits an async codebase with a modest task surface. Celery brings more brokers, more integrations, mature scheduling through beat and a large body of operational knowledge, at the cost of a much larger configuration surface. Pick ARQ for simplicity, Celery for breadth.

Do background workers share the FastAPI app's database pool? No. Workers are separate processes with their own event loops, so they build their own engine and pool at worker startup. Size those connections as part of the same database budget as the web tier, or a busy queue and a busy API will exhaust it together.

Why does moving from BackgroundTasks to a queue break my task arguments? Because BackgroundTasks passes the object itself in memory while a queue encodes every argument to bytes. An ORM row, a session, an open HTTP client or anything holding a closure has no encoded form, so the enqueue call raises where the in-process call never did.

Is Celery's async support good enough now? It works, but Celery's execution model is still fundamentally synchronous with worker pools, so an async-first codebase ends up bridging between the two. That bridging is the tax you pay for the ecosystem; if you are not collecting on the ecosystem, ARQ removes the tax.

Can I mix BackgroundTasks and a durable queue in one app? Yes, and it is usually the right answer. Keep genuinely optional work — a cache warm, an analytics ping — in BackgroundTasks where it costs nothing, and route anything whose loss costs money or trust to the queue. The mistake is using one mechanism for both classes of work.

Related Reading

• Up to the topic: Background Task Processing for how in-process tasks are dispatched.
• The failure that pushes people off BackgroundTasks: When BackgroundTasks silently fails.
• Implementing the ARQ choice: Running ARQ Workers with FastAPI.
• The rule any queue imposes: Retry and idempotency for tasks.
• Budgeting the worker tier's connections: Fixing asyncpg pool exhaustion.