FastAPI BackgroundTasks vs Celery vs ARQ

Key takeaways:

  • BackgroundTasks is in-process and fire-and-forget — fine only when losing a job is acceptable.
  • Celery and ARQ are durable queues with retries, scheduling, and independent scaling.
  • ARQ is async-native and lightweight; Celery is mature with a broad ecosystem.
  • Durable workers run as separate processes with their own database pools.
  • Whatever you choose, make jobs idempotent so retries are safe.

This comparison operationalizes Background Task Processing. Read that page for the in-process versus queued model.

The Problem This Solves

"Run this after responding" has three very different answers depending on whether the work can be lost, must be retried, or must scale independently. Choosing the wrong one means either dropped jobs or needless infrastructure.

Prerequisites

  • A FastAPI app with deferred work to run.
  • Redis available if you adopt ARQ or Celery-on-Redis.

The Comparison

CapabilityBackgroundTasksARQCelery
Durability across restartsNoYesYes
Automatic retriesNoYesYes
Scheduling (cron)NoYes (cron jobs)Yes (beat)
Separate scalingNoYesYes
Async-nativeRuns on the loopYesNewer support
Setup costNoneLowModerate

Step-by-Step: Choosing

1. Fire-and-forget, loss acceptable → BackgroundTasks

from fastapi import BackgroundTasks


@app.post("/audit")
async def audit(event: dict, tasks: BackgroundTasks) -> dict[str, str]:
    tasks.add_task(write_audit_line, event)   # Lost on crash — acceptable here.
    return {"status": "ok"}

2. Durable + async-native → ARQ

async def sync_invoice(ctx: dict, invoice_id: str) -> None:
    await push_to_accounting(invoice_id)      # Retried by ARQ on failure.


class WorkerSettings:
    functions = [sync_invoice]
    max_tries = 5

3. Durable + broad ecosystem / scheduling → Celery

from celery import Celery

celery = Celery("app", broker="redis://localhost:6379/0")


@celery.task(bind=True, max_retries=5)
def rebuild_search_index(self, tenant_id: str) -> None:
    # Mature retry/backoff and beat scheduling come built in.
    reindex(tenant_id)

Edge Cases and Gotchas

  • Result backends. If you need job results, configure a result backend; otherwise jobs are fire-and-forget even in a durable queue.
  • Serialization. Arguments are serialized to the broker; pass IDs, not large objects or ORM instances.
  • Context loss. Forward the correlation ID into the job for tracing, per Observability and Tracing.

Verification

def test_audit_does_not_block_response(client):
    resp = client.post("/audit", json={"type": "login"})
    assert resp.status_code == 200   # Returns without awaiting the task.

For durable queues, assert a job is enqueued and that re-running it is idempotent.