Managing Environment Variables with Pydantic Settings

Key takeaways:

  • Declare each variable as a typed field so the environment is parsed and validated, not merely read.
  • Matching is case-insensitive after the prefix is stripped — app_max_connections populates max_connections.
  • extra="forbid" catches misspelled keys in a .env file but not misspelled process environment variables. This is verified below.
  • Complex types are parsed as JSON, so a list field wants ["a","b"], not a,b.
  • Construct Settings once at startup, so a bad value stops the deploy rather than the first request that touches it.

This is the hands-on guide under Configuration Management. It covers how variables become fields; for choosing which file each environment reads and how credentials are injected, see Secrets and .env Files Per Environment.

The Problem This Solves

os.environ["DATABASE_URL"] has three defects and they compound. It returns a string, so MAX_CONNECTIONS arrives as "25" and the bug appears wherever someone forgot int(). It fails at the moment of access, so a variable read only by the export endpoint is missing for a week before anyone finds out. And the reads are scattered, so no single place in the codebase describes what the service needs to run.

A BaseSettings model fixes all three at once — but only if you know what it actually does with the environment, because several of its behaviours are not what the field declaration suggests. The specific one that costs people real incidents is at the end of this page.

Why It Happens

BaseSettings is a normal Pydantic model with an unusual way of getting its input. On construction it asks a list of settings sources, in priority order, for a dict of values, merges them, and validates the merged dict exactly as any Pydantic model validates a payload. The default sources, highest priority first, are: arguments passed to Settings(...), the process environment, the dotenv file, the secrets directory, and finally the field defaults.

Two consequences follow, and both explain behaviour that otherwise looks arbitrary.

First, validation happens after merging, not per source. So a value from a .env file and a value from the environment are indistinguishable by the time constraints are applied, and every field is validated in one pass — you get all the errors at once rather than the first one.

Second, the sources do not work the same way. The environment source performs lookups: for each field it knows about, it constructs the expected variable name and asks the environment for it. The dotenv source performs a scan: it reads the whole file and returns every key in it. That asymmetry is invisible until you turn on extra="forbid" and assume it protects you from typos.

Why extra=forbid catches file typos but not environment typosThe dotenv source reads every key in the file, so an unknown key reaches the model and is rejected. The environment source asks only for variables matching declared fields, so an unknown variable is never fetched and never rejected.Dotenv source — SCANS the fileAPP_NAME=realAPP_NAM=misspelledBoth keys are handed to the model.The unknown one hits extra=forbid.Result: ValidationErrorEnvironment source — LOOKS UP fieldsfor each field: getenv(APP_NAME)APP_NAM is never asked forThe model never learns it was set.extra=forbid has nothing to reject.Result: silently ignoredmerged dict → validationDeployed environments configure via the right-hand path, which is the unprotected one.

The Fix

1. Declare the model

class DatabaseConfig(BaseModel):
    host: str
    port: int = 5432
    pool_size: int = 10


class Settings(BaseSettings):
    model_config = SettingsConfigDict(
        env_prefix="APP_",
        env_file=ENV_FILE,
        env_nested_delimiter="__",
        case_sensitive=False,
        extra="forbid",
    )

    service_name: str
    environment: Literal["development", "staging", "production"]
    max_connections: int = Field(default=10, ge=1, le=100)
    request_timeout_s: float = 3.0          # nothing sets this: the default survives
    debug: bool = False
    feature_flags: list[str] = []
    api_key: SecretStr = SecretStr("unset")
    db: DatabaseConfig

Given a .env file containing APP_SERVICE_NAME, APP_ENVIRONMENT=development, APP_DEBUG=yes, a JSON array in APP_FEATURE_FLAGS, and APP_DB__HOST/APP_DB__PORT — plus a process environment supplying APP_ENVIRONMENT=production, APP_API_KEY, and a deliberately lower-case app_max_connections=80 — this is what the process actually resolves:

$ GET /resolved
200 OK
{
  "values": {
    "service_name": "checkout-from-dotenv",
    "environment": "production",
    "max_connections": 80,
    "request_timeout_s": 3.0,
    "debug": true,
    "feature_flags": [
      "beta-checkout",
      "new-pricing"
    ],
    "api_key": "**********",
    "db": {
      "host": "db.dev.internal",
      "port": 5432,
      "pool_size": 10
    }
  },
  "python_types": {
    "max_connections": "int",
    "request_timeout_s": "float",
    "debug": "bool",
    "feature_flags": "list",
    "db": "DatabaseConfig"
  },
  "where_each_value_came_from": {
    "service_name": ".env file (nothing in the environment set it)",
    "environment": "process environment, overriding development in the .env file",
    "max_connections": "process environment as lower-case app_max_connections",
    "request_timeout_s": "the class default; no variable of either kind set it",
    "db.pool_size": "the nested model's default; only DB__HOST and DB__PORT were set"
  }
}

Five separate behaviours are demonstrated there, none of them asserted:

  • Precedence is real. environment is "production" from the process environment even though the .env file said development. The environment source outranks the file.
  • Case does not matter. max_connections is 80, supplied as app_max_connections. After the prefix is stripped, matching is case-insensitive.
  • Defaults survive. request_timeout_s stayed 3.0 because nothing set it, and db.pool_size stayed 10 because only two of the three nested fields were supplied.
  • Types are real types. "yes" became the boolean True, "80" became an int, the JSON array became a list, and the nested block became a DatabaseConfig instance — not a dict.
  • Secrets are masked. api_key renders as **********. SecretStr overrides __str__ and __repr__, so an accidental log of the settings object cannot leak it; get_secret_value() is the only way out, and it greps.

2. Let bad values stop the deploy

Because validation runs over the merged dict, a broken environment produces one report of everything wrong:

$ GET /misconfiguration
200 OK
[
  {
    "case": "APP_ENVIRONMENT=prod  (not one of the allowed values)",
    "result": "ValidationError",
    "errors": [
      {
        "field": "environment",
        "type": "literal_error",
        "msg": "Input should be 'development', 'staging' or 'production'"
      }
    ]
  },
  {
    "case": "APP_MAX_CONNECTIONS=500  (outside the declared range)",
    "result": "ValidationError",
    "errors": [
      {
        "field": "max_connections",
        "type": "less_than_equal",
        "msg": "Input should be less than or equal to 100"
      }
    ]
  },
  {
    "case": "APP_MAX_CONNECTIONS=lots  (not an integer)",
    "result": "ValidationError",
    "errors": [
      {
        "field": "max_connections",
        "type": "int_parsing",
        "msg": "Input should be a valid integer, unable to parse string as an integer"
      }
    ]
  },
  {
    "case": "APP_SERVICE_NAMES=typo  (extra='forbid' catches the misspelling)",
    "result": "constructed without error"
  }
]

Literal catches the almost-right value, Field(ge=..., le=...) catches the out-of-range one, and the type itself catches the unparseable one. Each error names the field, the error type, and a message you can paste into an incident channel.

3. The one that will bite you: extra="forbid" and misspelled variables

Look at the last case above. APP_SERVICE_NAMES — a plural typo for service_name — constructed without error, with extra="forbid" set. The label on that case was written expecting the opposite, and the run corrected it.

This is worth isolating, because the advice to set extra="forbid" so that unknown variables fail at startup is repeated everywhere, and it is only half true:

$ GET /typo-detection
200 OK
[
  {
    "case": "unknown key PROBE_NAM in the .env file",
    "result": "ValidationError",
    "errors": [
      {
        "field": "probe_nam",
        "type": "extra_forbidden",
        "msg": "Extra inputs are not permitted"
      }
    ]
  },
  {
    "case": "unknown variable PROBE_NAM in the process environment",
    "result": "constructed: name='real'"
  }
]

The identical misspelling raises extra_forbidden from a .env file and is silently ignored from the process environment. The mechanism is the asymmetry described above: the dotenv source hands over every key it read, so an unknown one reaches the model and is refused. The environment source only ever looks up variables matching fields that exist, so PROBE_NAM is never fetched and the model never learns it was set.

The practical consequence is that extra="forbid" does not protect your production deploys, where configuration arrives as real environment variables. It protects developers editing .env files, which is worth having, but it is not the safety net it is usually sold as. If you need typo detection in deployed environments, the check has to be explicit — see the verification below.

Verification

Assert the two properties that matter, then close the gap extra="forbid" leaves:

def test_missing_required_variable_fails_fast(monkeypatch):
    monkeypatch.delenv("APP_DATABASE_URL", raising=False)
    with pytest.raises(ValidationError):
        Settings(_env_file=None)      # no file, no environment: required field missing


def test_environment_beats_the_env_file(monkeypatch, tmp_path):
    env_file = tmp_path / ".env"
    env_file.write_text("APP_ENVIRONMENT=development\n")
    monkeypatch.setenv("APP_ENVIRONMENT", "production")
    assert Settings(_env_file=env_file).environment == "production"


def test_no_unknown_app_variables_are_set():
    """The check extra='forbid' cannot do for the process environment."""
    known = {f"APP_{name.upper()}" for name in Settings.model_fields}
    known |= {f"APP_{n.upper()}__{s.upper()}"
              for n, m in Settings.model_fields.items()
              if hasattr(m.annotation, "model_fields")
              for s in m.annotation.model_fields}
    unknown = {k for k in os.environ if k.startswith("APP_")} - known
    assert not unknown, f"unknown APP_* variables set: {sorted(unknown)}"

The third test is the one worth adding today. Run it at startup rather than only in CI — as a one-line check inside create_app — and a mistyped variable name in a deployment manifest fails the rollout instead of quietly leaving the field at its default.

Logging the resolved configuration once at boot is the other half. Because SecretStr masks itself, settings.model_dump() is safe to emit, and it answers "what was this pod actually running" without a shell into the container.

Trade-offs and When Not To

Strict typing has a cost at the boundary with operators. A Literal field means an environment nobody anticipated — a new region name, a fourth deployment tier — cannot be introduced without a code change and a release. That is usually the correct trade, but be deliberate about which fields get closed types: constrain the ones where a wrong value is dangerous, and leave descriptive strings open.

The JSON parsing of complex types is the sharpest edge in daily use. Operators reasonably expect APP_FEATURE_FLAGS=a,b,c to work, and it produces a validation error that reads as if the variable were unset. If humans will edit the value, add a mode="before" validator that accepts both spellings rather than teaching everyone the JSON rule.

Finally, lru_cache on the accessor is standard advice and it interacts badly with tests: the first construction is memoised for the process, so a later monkeypatch.setenv has no effect. Either construct Settings directly in tests, as above, or call get_settings.cache_clear() in a fixture. The alternative — no caching at all — reparses and revalidates the whole environment on every request, which is wasted work on a hot path.

FAQ

How does pydantic-settings match an environment variable to a field? It lowercases the variable name, strips the configured env_prefix, and looks for a field with that name. Matching is case-insensitive by default, so a verified run shows a field named max_connections being populated by a variable spelled app_max_connections in lower case.

Does extra="forbid" catch a misspelled environment variable? Only in a .env file, not in the process environment. A verified run shows PROBE_NAM in a .env file raising extra_forbidden, while the identical PROBE_NAM exported as a real environment variable is silently ignored. The dotenv source reads every key in the file; the environment source only looks up the fields it knows about.

How do I configure a nested settings model? Give the parent a nested model field and set env_nested_delimiter, conventionally a double underscore. With env_prefix APP_ and delimiter __, the variables APP_DB__HOST and APP_DB__PORT populate a DatabaseConfig submodel, and any field of that submodel not supplied keeps its own default.

Why does my list field reject a comma-separated value? Complex types are parsed as JSON, so a list field expects the value to be a JSON array such as ["a","b"] rather than a,b. If operators need comma-separated input, add a field validator with mode="before" that splits the string before validation runs.

Where should Settings be constructed? Once, at import of a config module, or behind an lru_cache accessor. Constructing it inside a request handler reparses and revalidates the environment on every call, and it delays a misconfiguration error until traffic arrives instead of surfacing it at startup.