Instrumenting FastAPI with OpenTelemetry

Key takeaways:

  • A TracerProvider plus a span processor is the whole SDK setup; the exporter is swappable.
  • FastAPIInstrumentor opens one server span per matched request, named after the route template.
  • Manual spans nest inside it automatically — no parent needs to be passed around.
  • An exception leaving a span sets it to ERROR, even a deliberate HTTPException.
  • The server span is UNSET on a 404 and ERROR on a 500; only 5xx counts as server error.

This guide is the tracing implementation behind Observability and Tracing. It covers what OpenTelemetry emits and how the span tree is shaped; the sibling guides cover the numeric signal and the job of carrying one identifier through logs, spans and error reports.

The Problem This Solves

A request that takes four seconds tells you nothing about where the four seconds went. Logs give you moments; a trace gives you the shape. Distributed tracing records a span per unit of work, each with a parent, so the request becomes a timeline you can read: this handler waited three seconds on that query, which waited on that downstream service.

The awkward part in practice is not the concept but the wiring. The OpenTelemetry Python SDK has several moving pieces — provider, processor, exporter, propagator, sampler — and their interaction decides what actually lands in your backend.

Why It Happens: The SDK's Object Graph

Four objects do all the work, and understanding their split makes the configuration obvious rather than magic.

A tracer provider is the root. It owns the resource — the attributes identifying this service, most importantly service.name — and it decides sampling. Tracers are obtained from it, and every span a tracer starts inherits the provider's resource.

A span processor receives spans as they start and finish. SimpleSpanProcessor hands each finished span straight to the exporter, on the thread that finished it. BatchSpanProcessor puts it on a queue that a background thread drains. That single choice is the difference between export latency landing on your p99 and not.

An exporter turns spans into bytes for somewhere else: OTLP over gRPC or HTTP for a collector, console for debugging, in-memory for tests.

A propagator serializes the active span context into a carrier — for HTTP, the W3C traceparent header — and parses it back on the other side. It is what makes a trace distributed rather than per-service.

The instrumentation library sits on top of all of this. FastAPIInstrumentor installs ASGI middleware that opens a span when a request arrives and closes it when the response finishes.

How a span travels from a FastAPI request to a collectorA request enters instrumented FastAPI, which asks a tracer for a span. The tracer belongs to a tracer provider holding the service resource and sampler. Finished spans go to a batch span processor, which queues them for an exporter that sends them to a collector.HTTP requesttraceparent headerInstrumented appopens server spanTracerProviderresource + samplerManual spansnest as childrenBatchProcessorbackground queueSampler dropsnothing exportedCollectorvia OTLP

Prerequisites

  • opentelemetry-sdk and opentelemetry-instrumentation-fastapi. The output below was produced with opentelemetry-sdk 1.44.0 and the 0.65b0 instrumentation on FastAPI 0.139.2, Python 3.12.
  • For production, opentelemetry-exporter-otlp and a collector endpoint.

The Fix

The example below is complete and executed. It uses InMemorySpanExporter and SimpleSpanProcessor so the spans can be inspected inside the same process — swap in BatchSpanProcessor(OTLPSpanExporter()) and nothing about the span tree changes.

from fastapi import FastAPI, HTTPException, Request
from fastapi.responses import JSONResponse
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter

# A real SDK provider. Production swaps in an OTLP exporter and a BatchSpanProcessor; the span
# tree recorded here is the same either way.
EXPORTER = InMemorySpanExporter()
PROVIDER = TracerProvider(resource=Resource.create({"service.name": "orders-api"}))
PROVIDER.add_span_processor(SimpleSpanProcessor(EXPORTER))
# Tracers come from this provider directly rather than the global one, so several
# example apps can coexist in one process. In an app you would call set_tracer_provider once.
tracer = PROVIDER.get_tracer("orders")

app = FastAPI()


@app.exception_handler(RuntimeError)
async def unhandled(request: Request, exc: RuntimeError) -> JSONResponse:
    return JSONResponse(status_code=500, content={"detail": "internal error"})


@app.get("/orders/{order_id}")
async def read_order(order_id: int) -> dict[str, int]:
    # A manual span nests inside the span the instrumentation opened for the request.
    with tracer.start_as_current_span("load_order", attributes={"order.id": order_id}):
        with tracer.start_as_current_span("db.query"):
            pass
    return {"order_id": order_id}


@app.get("/orders/{order_id}/explode")
async def explode(order_id: int) -> dict[str, str]:
    with tracer.start_as_current_span("load_order", attributes={"order.id": order_id}):
        raise HTTPException(status_code=404, detail="no such order")


@app.get("/boom")
async def boom() -> dict[str, str]:
    with tracer.start_as_current_span("risky_work"):
        raise RuntimeError("downstream unavailable")


@app.get("/spans")
async def spans() -> dict[str, object]:
    """One compact line per recorded span: name, parent, status, and any events."""
    finished = EXPORTER.get_finished_spans()
    by_id = {s.get_span_context().span_id: s.name for s in finished}
    out = []
    for span in finished:
        parent = by_id.get(span.parent.span_id, "<remote>") if span.parent else "-"
        events = ",".join(e.name for e in span.events) or "-"
        out.append(
            f"{span.name}  | parent={parent}"
            f"  | status={span.status.status_code.name}  | events={events}"
        )
    return {"spans": out}


FastAPIInstrumentor.instrument_app(app, tracer_provider=PROVIDER)

Running it produces the following. This is the recorded output of that app, not a sketch of what spans usually look like:

$ GET /orders/7
200 OK
{
  "order_id": 7
}

$ GET /orders/9/explode
404 Not Found
{
  "detail": "no such order"
}

$ GET /boom
500 Internal Server Error
{
  "detail": "internal error"
}

$ GET /spans
200 OK
{
  "spans": [
    "db.query  | parent=load_order  | status=UNSET  | events=-",
    "load_order  | parent=GET /orders/{order_id}  | status=UNSET  | events=-",
    "GET /orders/{order_id} http send  | parent=GET /orders/{order_id}  | status=UNSET  | events=-",
    "GET /orders/{order_id} http send  | parent=GET /orders/{order_id}  | status=UNSET  | events=-",
    "GET /orders/{order_id}  | parent=-  | status=UNSET  | events=-",
    "load_order  | parent=GET /orders/{order_id}/explode  | status=ERROR  | events=exception",
    "GET /orders/{order_id}/explode http send  | parent=GET /orders/{order_id}/explode  | status=UNSET  | events=-",
    "GET /orders/{order_id}/explode http send  | parent=GET /orders/{order_id}/explode  | status=UNSET  | events=-",
    "GET /orders/{order_id}/explode  | parent=-  | status=UNSET  | events=-",
    "risky_work  | parent=GET /boom  | status=ERROR  | events=exception",
    "GET /boom http send  | parent=GET /boom  | status=ERROR  | events=-",
    "GET /boom http send  | parent=GET /boom  | status=UNSET  | events=-",
    "GET /boom  | parent=-  | status=ERROR  | events=-"
  ]
}

Five things in that output are worth stopping on.

Spans finish inside-out. db.query is recorded first and the server span last, because a processor sees a span when it ends. Backends reassemble the tree from parent ids, so ordering in the export stream is irrelevant — but it does mean a crashed process loses the outermost spans first.

Nesting is implicit. Nothing passed a parent anywhere. start_as_current_span reads the active span from a context variable and writes itself back, so load_order finds the server span and db.query finds load_order. This is also why nesting silently breaks when work moves to another thread.

The 404 is not a server error. GET /orders/{order_id}/explode has status UNSET, while GET /boom has ERROR. The specification is explicit that server spans are set to ERROR only for 5xx: a 404 means the server did its job correctly and the client asked for something absent. If you build an alert on span status, 4xx traffic will not appear in it.

But the manual span is ERROR. Inside explode, HTTPException propagated out of the with block, so the span recorded an exception event and flipped to ERROR. This is the most misleading thing in a freshly instrumented service: perfectly ordinary 404s light up your error views. The span sees an exception and cannot know your framework treats it as a routine response. Raise outside the span, or construct it with record_exception=False, set_status_on_exception=False.

There are two http send spans per request. The ASGI instrumentation spans each send event — http.response.start and http.response.body. Useful for a streaming endpoint where the body takes far longer than the headers, noise everywhere else.

The attributes you actually get

$ GET /server-span-attributes
200 OK
{
  "attributes": {
    "http.flavor": "1.1",
    "http.host": "testserver:None",
    "http.method": "GET",
    "http.route": "/orders/{order_id}",
    "http.scheme": "http",
    "http.server_name": "testserver",
    "http.status_code": 200,
    "http.target": "/orders/7",
    "http.url": "http://testserver/orders/7",
    "http.user_agent": "python-httpx/0.28.1",
    "net.peer.ip": "127.0.0.1",
    "net.peer.port": 123
  }
}

Note that http.route holds the template while http.target and http.url hold the concrete path. That distinction matters twice over: http.route is the safe grouping key, and http.url carries the query string, which is exactly where API keys and email addresses end up. Traces are usually retained for days and are readable by everyone with backend access, so treat span attributes as data you are publishing.

These are also the old HTTP semantic conventions — http.method and http.status_code rather than the stabilised http.request.method and http.response.status_code. This instrumentation version still emits the old names by default; setting OTEL_SEMCONV_STABILITY_OPT_IN=http switches to the new ones. Check which names your backend's dashboards expect before flipping it.

Context Propagation Across Services

A trace only spans services if the context travels with the request. The example below sends a fixed W3C traceparent in through a /selftest endpoint that issues real HTTP calls back into the same app, then reports what the app made of it. It reports relationships rather than ids, because ids are random on every run.

@app.get("/work")
async def work(request: Request) -> dict[str, object]:
    span = trace.get_current_span().get_span_context()
    # The headers the app would send to a downstream service, filled in by the propagator.
    outgoing: dict[str, str] = {}
    inject(outgoing)
    trace_id = format(span.trace_id, "032x")
    outgoing_trace_id = outgoing.get("traceparent", "").split("-")[1]
    return {
        "received_traceparent": request.headers.get("traceparent", "<none>"),
        "joined_upstream_trace": trace_id == UPSTREAM.split("-")[1],
        "outgoing_header_sent": bool(outgoing_trace_id),
        "outgoing_continues_this_trace": outgoing_trace_id == trace_id,
        "sampled": span.trace_flags.sampled,
    }

$ GET /selftest
200 OK
{
  "upstream_traceparent": "00-11111111111111111111111111111111-2222222222222222-01",
  "with_incoming_context": {
    "received_traceparent": "00-11111111111111111111111111111111-2222222222222222-01",
    "joined_upstream_trace": true,
    "outgoing_header_sent": true,
    "outgoing_continues_this_trace": true,
    "sampled": true
  },
  "without_incoming_context": {
    "received_traceparent": "<none>",
    "joined_upstream_trace": false,
    "outgoing_header_sent": true,
    "outgoing_continues_this_trace": true,
    "sampled": true
  }
}

With a traceparent present the app adopted the caller's trace id; without one it minted a fresh trace. In both cases inject() produced a header continuing whichever trace was active. That is the whole propagation contract, and it explains why an instrumented HTTP client needs no configuration to keep a trace intact — and equally why work handed to a queue or a thread arrives with nothing unless you inject the context into the message yourself.

Verification

The in-memory exporter is not just a documentation trick; it is the right way to test instrumentation. Assert on structure, never on timing:

def test_order_span_tree(client, exporter):
    client.get("/orders/7")
    spans = {s.name: s for s in exporter.get_finished_spans()}
    assert "load_order" in spans
    assert spans["load_order"].attributes["order.id"] == 7
    # The manual span really is a child of the request span.
    parent = spans["GET /orders/{order_id}"].get_span_context().span_id
    assert spans["load_order"].parent.span_id == parent

A test like this catches the failure that is otherwise invisible until production: someone moves a call into a thread or a task, the span quietly becomes a root, and one trace silently becomes two.

In a running service the fastest check is a ConsoleSpanExporter for one minute. If spans appear with the right service.name and the right parents, the SDK is wired correctly. If nothing appears at all, the cause is almost always ordering — the provider was configured after the instrumentation ran, or the app object was created before configure_tracing() was called.

Trade-offs and When Not To

Batching trades freshness for latency. BatchSpanProcessor is right in production, but a span may sit in the queue for seconds, and queued spans are lost if the process is killed. During an incident that gap is confusing; calling force_flush() in a shutdown hook narrows it.

Auto-instrumentation is not free. Every request allocates spans and copies attributes, even for traces that are later dropped, unless the sampling decision is made at the root. Under high load, sample.

Manual spans stop paying off quickly. A span per function produces traces nobody can read. Instrument boundaries — network calls, queue handoffs, expensive computations — and let the flame graph stay legible.

Do not instrument to replace metrics. Traces are sampled, so they cannot answer "what is my error rate". That is what Prometheus metrics are for. Traces explain the why behind a number that metrics already gave you.

FAQ

Why is my server span status UNSET even though the request returned 404? OpenTelemetry only marks server spans as ERROR for 5xx responses. A 4xx is treated as a successful outcome of the server's job, because the fault lies with the request. Filter on the status code attribute rather than span status when you want to find client errors.

Why does a manual span show ERROR when the endpoint only raised HTTPException? Any exception leaving a start_as_current_span block sets that span to ERROR and records an exception event, and HTTPException is an ordinary Python exception. Raise it outside the span, or pass record_exception and set_status_on_exception as False, if a 404 is a normal outcome.

Do I need to propagate trace context manually between services? No, for HTTP calls made with an instrumented client. The instrumentation extracts the incoming traceparent header and the propagator injects it into outbound requests, so the trace continues across services. You only inject manually when you hand work to a transport the instrumentation does not cover, such as a queue message.

What are those extra http send spans in my trace? The ASGI instrumentation opens a child span for each ASGI send event, so a normal response produces two: one for the response start and one for the body. They are useful for spotting slow streaming responses, and can be turned off when they only add noise.

Should I export spans synchronously? Not in production. A SimpleSpanProcessor exports on the request path and adds its latency to every response. Use BatchSpanProcessor so spans queue in memory and flush on a background thread, and keep a simple processor only for tests where determinism matters more than throughput.