Running Sync Code in a Threadpool in FastAPI

Key takeaways:

  • A plain def endpoint is already dispatched to a threadpool by Starlette. You do not need to wrap anything inside it.
  • Inside async def, use run_in_threadpool or anyio.to_thread.run_sync — they are the same mechanism and share one pool.
  • The pool is governed by an AnyIO capacity limiter whose default total_tokens is 40 per event loop.
  • Resize the limiter from inside the loop (a lifespan handler), never at import time.
  • Threads fix responsiveness, not throughput. CPU-bound work still needs a process pool.

This is the mechanical companion to Async Correctness and Concurrency. That page explains why a blocked loop stalls a whole worker; this one explains exactly where the sync work goes instead, and shows the wall-clock evidence.

How FastAPI dispatches async and sync endpoints An async def endpoint runs directly on the event loop thread. A plain def endpoint and any run_in_threadpool call both pass through the AnyIO capacity limiter, which allows forty concurrent worker threads by default. Where the body of an endpoint actually runs async def endpoint stays on the loop plain def endpoint run_in_threadpool() capacity limiter total_tokens = 40 worker thread 1 worker thread 2 ... up to 40 Both sync paths share one pool, so they compete for the same 40 tokens.
Only async def bodies execute on the loop thread. Everything sync funnels through a single, bounded pool.

The Problem This Solves

You have a function you cannot make async — a vendor SDK that only ships a sync client, a legacy DB-API driver, an image resize, a PDF render. It has to be called from a FastAPI request. The question is not whether to run it in a thread but which of the three available dispatch routes to use, and what the ceiling on that route is when a hundred requests arrive at once.

Most of the confusion here comes from a single fact that is easy to miss: FastAPI already threads sync endpoints for you. Developers who do not know this either wrap things twice, or convert a perfectly healthy def endpoint to async def for consistency and accidentally move blocking work onto the loop.

Why It Happens: Starlette's Dispatch Decision

When you register a path operation, FastAPI inspects the callable with asyncio.iscoroutinefunction. If it is a coroutine function, the endpoint is awaited directly on the event loop. If it is not, Starlette wraps it so that the whole body is submitted to a worker thread and awaited from the loop. That decision is made once, at route-registration time, and it applies to the entire function body — not just to the blocking line inside it.

run_in_threadpool is the same machinery exposed as a function. In current Starlette it is a thin wrapper: it binds keyword arguments with functools.partial and hands the callable to anyio.to_thread.run_sync. There is no separate Starlette-owned pool. Both routes end up in AnyIO's worker-thread machinery, and both are gated by the same object: the default thread limiter, an anyio.CapacityLimiter stored per event loop, whose total_tokens defaults to 40.

"40 threads" is worth reading precisely. It is not a pre-spawned pool of 40 OS threads; it is a cap on how many run_sync calls may be in flight simultaneously. AnyIO creates threads lazily, idles them for a short while, and reuses them. The forty-first concurrent call does not fail — it waits for a token, which is exactly the queuing behaviour that turns "we offloaded it" into "we moved the bottleneck".

The Fix: Pick the Right Route and Measure It

The example below makes the dispatch behaviour observable. Each endpoint does the same 200 ms time.sleep, and each /probe/* endpoint fires five concurrent in-process requests at one of them and returns the measured wall clock alongside the number of distinct OS threads that served them.

"""Measure how Starlette's threadpool actually runs sync work."""
import asyncio
import threading
import time

import anyio
import anyio.to_thread
import httpx
from fastapi import FastAPI
from starlette.concurrency import run_in_threadpool

app = FastAPI()

SLEEP = 0.20


def blocking_work(tag: str) -> str:
    """A synchronous call that holds its thread for SLEEP seconds."""
    time.sleep(SLEEP)
    return f"{tag}@{threading.current_thread().name}#{threading.get_ident()}"


@app.get("/sync-endpoint")
def sync_endpoint() -> dict[str, str]:
    """A PLAIN def endpoint. Starlette dispatches it to the threadpool for us."""
    return {"thread": blocking_work("sync-endpoint")}


@app.get("/async-endpoint-blocking")
async def async_endpoint_blocking() -> dict[str, str]:
    """The bug: sync work called directly from the loop thread."""
    return {"thread": blocking_work("async-blocking")}


@app.get("/async-endpoint-offloaded")
async def async_endpoint_offloaded() -> dict[str, str]:
    """The fix: the same sync work handed to the threadpool."""
    return {"thread": await run_in_threadpool(blocking_work, "offloaded")}


async def fan_out(path: str, n: int) -> dict:
    """Fire n concurrent in-process requests at our own app and time the whole batch."""
    transport = httpx.ASGITransport(app=app)
    async with httpx.AsyncClient(transport=transport, base_url="http://probe") as client:
        started = time.perf_counter()
        responses = await asyncio.gather(*[client.get(path) for _ in range(n)])
        elapsed = time.perf_counter() - started
    threads = {r.json()["thread"].split("#")[1] for r in responses}
    return {
        "path": path,
        "requests": n,
        "elapsed_s": round(elapsed, 1),
        "serial_would_be_s": round(n * SLEEP, 1),
        "distinct_threads": len(threads),
    }


@app.get("/limiter")
async def limiter() -> dict[str, float]:
    """Report the live capacity of the threadpool this worker will use."""
    lim = anyio.to_thread.current_default_thread_limiter()
    return {"total_tokens": lim.total_tokens, "borrowed_tokens": lim.borrowed_tokens}

Running it under the verification harness produces this — every number below is a real measurement taken by the process, not an estimate:

$ GET /limiter
200 OK
{
  "total_tokens": 40.0,
  "borrowed_tokens": 0.0
}

$ GET /probe/sync-endpoint
200 OK
{
  "path": "/sync-endpoint",
  "requests": 5,
  "elapsed_s": 0.2,
  "serial_would_be_s": 1.0,
  "distinct_threads": 5
}

$ GET /probe/async-blocking
200 OK
{
  "path": "/async-endpoint-blocking",
  "requests": 5,
  "elapsed_s": 1.0,
  "serial_would_be_s": 1.0,
  "distinct_threads": 1
}

$ GET /probe/offloaded
200 OK
{
  "path": "/async-endpoint-offloaded",
  "requests": 5,
  "elapsed_s": 0.2,
  "serial_would_be_s": 1.0,
  "distinct_threads": 5
}

Read the three probes together, because the contrast is the whole lesson:

  • /sync-endpoint — a plain def with no offloading code anywhere in it. Five concurrent requests finished in 0.2 s and were served by five distinct threads. Nobody wrote run_in_threadpool; Starlette did the dispatch.
  • /async-endpoint-blocking — the identical time.sleep, but inside async def. Five requests took 1.0 s, exactly the serial total, on one thread. That single thread is the event loop, and while it slept nothing else on the worker could run.
  • /async-endpoint-offloaded — the same async def with the call wrapped. Back to 0.2 s across five threads.

The middle case is the one that reaches production. It is not slow because threads are fast; it is slow because there is only one loop thread and it was asleep.

The Two Helpers Are One Helper

starlette.concurrency.run_in_threadpool and anyio.to_thread.run_sync are frequently discussed as if they were alternatives with different characteristics. They are not.

@app.get("/to-thread-vs-run-in-threadpool")
async def compare_helpers() -> dict[str, str]:
    """Both helpers land on the same anyio worker-thread pool."""
    a = await anyio.to_thread.run_sync(blocking_work, "anyio.to_thread")
    b = await run_in_threadpool(blocking_work, "run_in_threadpool")
    return {
        "anyio": a.split("#")[0],
        "starlette": b.split("#")[0],
        "same_thread_reused": str(a.split("#")[1] == b.split("#")[1]),
    }

$ GET /to-thread-vs-run-in-threadpool
200 OK
{
  "anyio": "anyio.to_thread@AnyIO worker thread",
  "starlette": "run_in_threadpool@AnyIO worker thread",
  "same_thread_reused": "True"
}

Both calls landed on a thread named AnyIO worker thread, and same_thread_reused is True because the second call reused the thread the first one had just released. The practical differences are ergonomic: run_in_threadpool accepts keyword arguments, while to_thread.run_sync takes positional arguments only and exposes cancellable and limiter parameters. Use run_in_threadpool in Starlette-flavoured code; reach for anyio.to_thread.run_sync when you want to pass your own limiter.

Resizing the Limiter — and Proving It Binds

Because every sync path shares one limiter, its size is a global property of the worker. Raising it lets more blocking calls overlap; lowering it protects a fragile downstream or caps memory. The following endpoint shrinks the limiter to two tokens, fans six requests at the def endpoint, and restores it:

@app.get("/probe/limiter-shrunk")
async def probe_limiter_shrunk() -> dict:
    """Shrink the limiter to 2 threads, then fan out 6 requests at the def endpoint."""
    lim = anyio.to_thread.current_default_thread_limiter()
    original = lim.total_tokens
    lim.total_tokens = 2
    try:
        result = await fan_out("/sync-endpoint", 6)
    finally:
        lim.total_tokens = original
    result["total_tokens"] = 2
    return result

$ GET /probe/limiter-shrunk
200 OK
{
  "path": "/sync-endpoint",
  "requests": 6,
  "elapsed_s": 0.6,
  "serial_would_be_s": 1.2,
  "distinct_threads": 2,
  "total_tokens": 2
}

Six 200 ms calls through two tokens took 0.6 s and used exactly two threads — three sequential batches of two. This is precisely what threadpool saturation looks like in production, just in miniature: the work is still off the loop, so the app stays responsive, but the sync endpoint's own latency is now three times its service time.

In a real app, set this once during startup rather than per request:

from contextlib import asynccontextmanager

import anyio.to_thread
from fastapi import FastAPI


@asynccontextmanager
async def lifespan(app: FastAPI):
    # Runs inside the loop, so the limiter it fetches is the one requests will use.
    limiter = anyio.to_thread.current_default_thread_limiter()
    limiter.total_tokens = 80
    yield


app = FastAPI(lifespan=lifespan)

The limiter is stored in an AnyIO run-scoped variable, which means it does not exist until an event loop is running. Setting it at module import time either raises or configures a limiter no request will ever see — a lifespan handler is the correct place, and it composes with the other startup work described in Async Database Sessions.

Verification

Two checks are worth keeping permanently.

First, assert that concurrency is real rather than assumed, by timing a fan-out:

import asyncio
import time

import httpx


async def test_sync_endpoint_is_actually_threaded():
    transport = httpx.ASGITransport(app=app)
    async with httpx.AsyncClient(transport=transport, base_url="http://t") as c:
        started = time.perf_counter()
        await asyncio.gather(*[c.get("/sync-endpoint") for _ in range(5)])
        # Serial would be 5 x 200ms; anything near 1s means the loop was blocked.
        assert time.perf_counter() - started < 0.5

Second, assert the limiter is configured as you intended, since a stray import-time tweak is invisible otherwise:

import anyio.to_thread


async def test_limiter_is_sized_for_this_service():
    assert anyio.to_thread.current_default_thread_limiter().total_tokens == 80

In production, the metric to graph is borrowed_tokens against total_tokens. When borrowed sits at the ceiling, sync requests are queuing for a thread and your p99 is being set by the limiter rather than by the downstream. Wiring that into your exporter is covered in Observability and Tracing.

Trade-offs and When Not To

Threads do not create CPU parallelism. The GIL still serialises pure Python bytecode. Offloading a CPU-bound function protects the loop — other requests keep being served — but the computation itself gets no faster, and forty concurrent CPU-bound threads will thrash. Send that work to a ProcessPoolExecutor, as covered in Fixing Blocking Calls in Async Routes.

A shared limiter means one endpoint can starve another. Because the pool is global to the worker, a slow sync report endpoint holding thirty tokens leaves ten for everything else. If two workloads have genuinely different capacity needs, give the risky one its own limiter and pass it explicitly:

import anyio

_reports = anyio.CapacityLimiter(4)


async def render_report(spec: dict) -> bytes:
    # Reports can never occupy more than 4 threads, whatever the default limiter allows.
    return await anyio.to_thread.run_sync(_render_sync, spec, limiter=_reports)

Raising total_tokens is not free. Each concurrent call is a real OS thread with a real stack, and if the sync work holds a database connection, thread count effectively becomes connection count. Raise the limiter and the pool together, or you will simply relocate the queue to the database, which is the failure described in Async Database Sessions.

Sometimes the right answer is not a thread at all. If the blocking work is slow and the caller does not need the result, it belongs in Background Task Processing rather than in a request-scoped thread.

FAQ

Do I need run_in_threadpool inside a plain def endpoint? No. Starlette already runs the entire body of a def endpoint in the threadpool, so wrapping calls inside it adds a second hop for no benefit. Only reach for run_in_threadpool when you are inside an async def and need to call something blocking.

What is the difference between run_in_threadpool and anyio.to_thread.run_sync? There is almost none. Starlette's run_in_threadpool is a thin wrapper that forwards to anyio.to_thread.run_sync after binding keyword arguments with functools.partial. Both draw from the same pool of AnyIO worker threads and both respect the same capacity limiter.

How many threads does FastAPI use for sync code? AnyIO's default capacity limiter allows 40 concurrent worker threads per event loop. That number is a limit on concurrency rather than a fixed pool size, and threads are created on demand and reused.

How do I change the threadpool size? Set total_tokens on the limiter returned by anyio.to_thread.current_default_thread_limiter, from inside the running event loop. The limiter is scoped to the loop, so do it in a lifespan handler rather than at import time.

Does moving CPU-bound work to a thread help? It keeps the event loop responsive, which protects other requests, but it does not make the CPU work itself faster because the GIL still serialises pure Python bytecode. For genuine CPU parallelism use a process pool.

Why did my endpoint get slower after I made it async def? Because a def endpoint runs in the threadpool where blocking is harmless, while an async def endpoint runs directly on the event loop where the same blocking call freezes every other request on that worker. Changing def to async def without also making the I/O async converts safe blocking into loop-blocking.