Observability and Tracing in FastAPI

Observability is the ability to answer new questions about a running system without deploying new code. In practice that comes down to three signals — structured logs, metrics, and distributed traces — and one identifier that ties them together.

This topic is part of Async, Background Tasks and Observability. It builds on the correlation ID assigned in middleware and on the consistent error envelope, turning per-request context into something you can query during an incident.

The four guides beneath this page each take one piece: instrumenting FastAPI with OpenTelemetry for traces, structured JSON logging with request IDs for the log substrate, Prometheus metrics for FastAPI for the numbers you alert on, and correlating logs, traces and errors for keeping one identifier alive across all of them.

What Each Signal Is Actually For

The three signals are usually presented as a set to collect. It is more useful to think of them as answers to three different questions asked at three different moments of an incident.

Metrics answer "is something wrong, and how widely?" They are aggregated numbers — counters and histograms — recorded for every request and stored cheaply because the aggregation throws away individual detail. That is what makes them suitable for alerting: a 5% error rate is a 5% error rate whether you served ten requests or ten million. It is also what makes them useless for diagnosis. A metric can tell you that POST /orders is failing 5% of the time; it can never tell you which orders or why, because that information was discarded at aggregation.

Traces answer "where did the time go, and which dependency caused it?" A trace is one request's path across services, broken into spans with parents and durations. Traces are sampled — you keep a fraction — because storing a full trace for every request costs far more than storing a counter. Sampling is why traces cannot be counted: five errors in your trace store might be five errors or five hundred.

Logs answer "what did this specific code decide at this specific moment?" They are the only signal that carries arbitrary context — the branch taken, the value compared, the exception message. They are also the most expensive per unit of information, which is why the discipline is to log decisions and identifiers rather than narration.

The failure mode when a team collects all three without this distinction is predictable: alerts fire on traces (unreliable, because sampled), dashboards are built from logs (expensive, because unaggregated), and nobody can answer why a particular request was slow (no traces retained). Collecting the signals is easy; using each for what it is good at is the part that takes deliberate design.

Prerequisites

Before instrumenting anything, three things should already be true, because each of them is a prerequisite for the signals being useful rather than merely present.

A request identifier exists and is assigned early. Middleware that reads an inbound X-Request-ID or mints one, covered in implementing custom middleware for request tracing. Without it the three signals are three disconnected datasets.

Errors have a consistent shape. If every handler invents its own error body, no metric can classify failures and no log query can find them. Global exception handlers come first.

The app is built by a factory. Instrumentation attaches to an app object, and several instrumentations must run before the app serves its first request. An app constructed at import time in a module that also configures telemetry produces ordering bugs that are miserable to diagnose — the classic symptom being spans that appear locally and vanish in production, where the import order differs.

Core Mechanics: How the Signals Are Emitted

All three signals are produced by the same mechanism in FastAPI: something wraps the request and records what happened. The differences are in where the wrapping sits and what it keeps.

Metrics and traces are typically emitted by ASGI middleware, which sees every request that reaches the application, including ones that never match a route. That is why a metrics middleware must be careful with its labels — an unmatched path is attacker-controlled, and using it as a label value is how a metrics backend gets flooded with millions of series. The Prometheus guide treats that as its central problem.

Logs are emitted from anywhere in your code, which is exactly why they need ambient context. A log line deep in a service function has no access to the request object, so the request id must reach it through a context variable rather than through a parameter. Python's contextvars propagate across await but not across a thread hop, which is the mechanism behind most "why is my request id missing" reports and the subject of correlating logs, traces and errors.

Traces add one thing the other two do not have: propagation. A span context is serialized into the traceparent header on every outbound call and parsed on the way in, so a trace continues across service boundaries. Metrics and logs have no equivalent — they are joined after the fact, by the identifier you put on them.

Production Implementation

Two decisions dominate what your observability actually costs and how well it works. Both are configuration rather than code, and both are usually wrong by default.

Sampling is decided once, at the root

With the default ParentBased sampler, a service samples the traces it starts and otherwise defers to whatever the caller decided. That decision travels in the traceparent flags. The consequence is worth stating plainly: a downstream service cannot record a trace its caller dropped, and cannot drop one its caller kept.

from opentelemetry.sdk.trace.sampling import ALWAYS_ON, ParentBased

EXPORTER = InMemorySpanExporter()
# ParentBased(ALWAYS_ON): sample every trace we start, but defer to the caller when there is one.
PROVIDER = TracerProvider(
    resource=Resource.create({"service.name": "orders-api"}),
    sampler=ParentBased(root=ALWAYS_ON),
)
PROVIDER.add_span_processor(SimpleSpanProcessor(EXPORTER))

SAMPLED = f"00-{TRACE}-{SPAN}-01"      # Upstream decided: record this trace.
NOT_SAMPLED = f"00-{TRACE}-{SPAN}-00"  # Upstream decided: drop this trace.


@app.get("/work")
async def work() -> dict[str, object]:
    with tracer.start_as_current_span("expensive_step") as span:
        ctx = span.get_span_context()
        return {
            "span_is_recording": span.is_recording(),
            "sampled_flag_set": bool(ctx.trace_flags.sampled),
        }

Sending the same request with each flag, and counting what reached the exporter, gives the real behaviour:

$ GET /selftest
200 OK
{
  "spans_before": 0,
  "results": {
    "upstream_sampled": {
      "span_is_recording": true,
      "sampled_flag_set": true,
      "spans_exported": 4
    },
    "upstream_not_sampled": {
      "span_is_recording": false,
      "sampled_flag_set": false,
      "spans_exported": 0
    },
    "no_upstream_context": {
      "span_is_recording": true,
      "sampled_flag_set": true,
      "spans_exported": 4
    }
  }
}

Zero spans exported for the dropped trace, and is_recording() reporting false inside the handler. That second detail is the practically useful one: you can check is_recording() before doing expensive work purely for telemetry — serializing a payload into a span attribute, for instance — and skip it entirely when the trace will be discarded.

It also explains a common confusion. If your service's sampling rate appears to be ignored, the traces are almost certainly arriving with a decision already made by an upstream service or a gateway. Sampling policy for a request path belongs wherever that path starts.

Probe traffic is the bulk of your telemetry

Liveness and readiness probes, metrics scrapes and uptime checks run on timers, not on user demand. On a low-traffic service they outnumber real requests by a wide margin, and by default the instrumentation traces all of them.

kwargs = {"excluded_urls": excluded} if excluded else {}
FastAPIInstrumentor.instrument_app(sub, tracer_provider=provider, **kwargs)

Two identical apps, one with excluded_urls="health,metrics" and one without, receiving eight probe requests for every two real ones:

$ GET /selftest
200 OK
{
  "requests_sent": {
    "probes": 8,
    "real": 2
  },
  "results": {
    "no_exclusions": {
      "server_spans": 10,
      "spans_for_probes": 8,
      "spans_for_real_traffic": 2
    },
    "health_and_metrics_excluded": {
      "server_spans": 2,
      "spans_for_probes": 0,
      "spans_for_real_traffic": 2
    }
  }
}

Eighty percent of the spans were noise, and the ratio in a real deployment is often worse — a one-second liveness probe generates 86,400 spans a day per instance regardless of whether anyone used the API. Excluding at the instrumentation level rather than filtering in the collector means the spans are never allocated, so you save the CPU as well as the storage.

The same reasoning applies to the other two signals. Probe requests should not increment your request counter, or your error rate becomes meaningless the moment a probe fails. They should not produce log lines above DEBUG either.

Async and Performance Notes

Telemetry runs on every request, so anything it does synchronously is added to every response.

Export in the background. A BatchSpanProcessor queues spans and flushes on a separate thread; a SimpleSpanProcessor exports inline. The simple processor is correct for tests, where determinism matters, and wrong for production, where it puts network latency on your request path. The same applies to log handlers: a handler writing to a network destination synchronously will eventually block the event loop when that destination is slow, which converts an observability outage into an application outage.

Watch for blocking inside the formatter. A JSON log formatter doing a DNS lookup or reading a file per record is blocking the event loop, and this is easy to introduce accidentally through a "helpful" enrichment step. Formatters should be pure functions of the record.

Bound cardinality, not volume. The expensive mistake in metrics is not recording too often but recording too many distinct series. A histogram labelled by route template is a handful of series; the same histogram labelled by raw URL is unbounded and attacker-controlled.

Check is_recording() before expensive attributes. As the sampling output above shows, an unsampled span is a cheap no-op object, but the code computing attributes for it still runs unless you guard it.

Testing Strategy

Observability code has a bad habit of being untested and then failing silently, which is the worst combination — you find out during the incident that the thing meant to help with incidents does not work.

Assert on structure, never on timing or ids. Span ids and durations differ on every run. Assert that a span exists, that its parent is the span you expect, and that the attribute you rely on is present. The OpenTelemetry guide shows the in-memory exporter this needs.

Test the join, not the individual signals. The valuable test asserts that a request with a known id produces a log line carrying that id and a span carrying it, because that is the property you depend on and the one that breaks when someone offloads work to a thread.

Use dependency_overrides to force the error paths. Error-rate metrics and error logging are the parts most likely to be wrong, precisely because they run least often. Overriding a dependency to raise is the cheapest way to exercise them:

def test_failures_are_counted_and_logged(client, captured):
    app.dependency_overrides[get_service] = lambda: FailingService()
    resp = client.get("/orders/1", headers={"x-request-id": "rid-1"})
    assert resp.status_code == 500
    assert any(r["request_id"] == "rid-1" and r["level"] == "ERROR" for r in captured.records)

Pin the metric names. A dashboard and an alert both depend on a metric name that nothing in the application enforces. A test asserting the exact names appear in the exposition output stops a rename from silently blinding your alerting.

Failure Modes and Diagnosis

Spans appear locally but not in production. Nearly always ordering: the tracer provider is configured after the app object is created, or after the instrumentation ran. Diagnose by logging whether the provider is the SDK's TracerProvider or the API's no-op default at startup.

The request id is missing from some log lines. The lines were emitted outside the request context — during startup, from a background task, or from code running in a threadpool. A context variable does not cross a thread hop. Diagnose by checking whether the affected lines all come from sync def endpoints or from library loggers, and see correlating logs, traces and errors.

A trace stops at a service boundary. The outbound call was not instrumented, so no traceparent was injected, and the downstream service started a fresh trace. Diagnose by logging the outbound headers of one request; if traceparent is absent, the client is the problem, not the receiver.

The metrics backend is overwhelmed. A label is unbounded — a user id, a raw path, an error message. Diagnose by counting distinct series per metric name; the offender is usually obvious and always recently added.

Latency jumped after adding instrumentation. A synchronous exporter, or a log handler writing over the network inline. Diagnose by switching to a console exporter briefly: if the latency disappears, it was export, not instrumentation.

Traces exist but are useless during incidents. Every request is one span with no children. Auto-instrumentation alone produces this. The fix is manual spans at the boundaries that matter — database calls, outbound requests, expensive computation.

Choosing Where to Spend Effort

The ordering in that last column is deliberate. Logs first, because the request id they carry is what everything else joins on and because they need no extra infrastructure. Metrics second, because alerting is what wakes you up and it must be reliable. Traces third — they are the most impressive and the least useful when you do not yet know that something is wrong.

FAQ

What are the three signals of observability and how do they differ? Metrics are aggregated numbers that tell you something is wrong and how widely. Traces show where one request spent its time and which dependency caused it. Logs record what a specific piece of code decided at a specific moment. They answer different questions and none of them substitutes for another.

Which signal should I add first? Structured logs with a request identifier, because every later signal keys off that identifier and logs are the only one that works without extra infrastructure. Metrics come second because they drive alerting, and traces third because they are most useful once you already know something is wrong.

Why can traces not replace metrics? Traces are sampled, so a count derived from them is a count of the sample rather than of reality. Error rate, request rate and latency percentiles must come from metrics, which aggregate every request. Traces then explain the requests behind a number metrics already gave you.

Does sampling decide anything per service? No, with the default parent-based sampler the root service decides and every downstream service inherits that decision through the traceparent header. A service cannot rescue a trace its caller dropped, which is what stops a distributed trace ending up half recorded.

Should health checks be traced? No. A liveness probe every second produces vastly more spans than real traffic and buys nothing, since a failing probe is already visible to your orchestrator. Exclude probe paths at the instrumentation level so the spans are never created.

How much does observability cost to run? The CPU cost of emitting the signals is usually small; the storage and ingestion cost is what surprises teams. Both scale with volume rather than with usefulness, which is why sampling traces, bounding metric label cardinality, and excluding probe traffic matter more than any in-process optimization.

Related Reading

• Up to the section: Async, Background Tasks and Observability.
• Traces: Instrumenting FastAPI with OpenTelemetry — the SDK's object graph, the real span tree, and context propagation.
• Logs: Structured JSON logging with request IDs — the formatter, and the reserved LogRecord names that break it.
• Metrics: Prometheus metrics for FastAPI — RED metrics, histogram buckets, and bounding label cardinality.
• The join key: Correlating logs, traces and errors — where context propagation survives and where it silently does not.
• Composes with: Middleware Implementation for assigning the id, and Error Handling and Global Exceptions for the shape of a failure.