---
name: airflow3-dlt-credentials
description: Reference for how credentials flow to dlt connectors in the Airflow 3 template pattern — Airflow Connections, secrets.toml fallback, container env vars, the secrets-toml-in-a-Variable flatten pattern, and the three-tier production model. Use when configuring destination or source credentials (incl. BigQuery), deciding where a secret belongs (Connection vs GitHub Actions vs environment file), debugging credential resolution, or reviewing the Fernet key setup.
---

# Airflow 3 dlt Credential Model

How credentials reach dlt connector subprocesses in the [airflow3-best-practice](https://github.com/Gemma-Analytics/airflow3-best-practice) template pattern, and where each kind of secret belongs in production.

## The three-tier production model

| Tier | Examples | Where it lives |
|---|---|---|
| Data credentials | DWH destination, source API tokens, source DB logins | **Airflow Connections/Variables** — stored in the metadata DB, masked in UI/logs |
| Infrastructure secrets | `AIRFLOW__CORE__FERNET_KEY`, API-server secret key, external metadata-DB `AIRFLOW__DATABASE__SQL_ALCHEMY_CONN` | **GitHub Actions secrets → deploy workflow → compose interpolation** — never in a file on the server, never in the repo |
| Non-secret bootstrap | `AIRFLOW_UID`, `AIRFLOW_PROJ_DIR`, `HIDE_SENSITIVE_VAR_CONN_FIELDS` | Server's environment file — not sensitive; keeps manual `docker compose` usable |

## Resolution precedence

For any dlt credential the effective order is:

**UI Connection > container env vars > `secrets.toml`**

- The utility helpers inject Connection values into the subprocess env *last*, so they win
- dlt's own provider chain puts env vars above its TOML files
- The mounted `~/.dlt/secrets.toml` (repo's `.dlt/` folder) is the file-based fallback — good for local dev, optional in production

## Path 1: Airflow Connections (data credentials — preferred)

**Destination** — Variable `dlt_destination` selects the destination; Connection `dlt_<destination>` supplies credentials via `get_destination_env()` from `dags/utils/dlt.py`:

- `dlt_postgres` (type Postgres): Host / Schema=database / Login / Password / Port
- `dlt_snowflake` (type Generic): Host=account identifier, Schema=database, Login, Password *or* key auth via Extra JSON (`warehouse`, `role`, `private_key`, `private_key_passphrase`)
- **No BigQuery branch exists** in `get_destination_env()` — BigQuery repos (typically service-account JSON) use Path 4 below, or add the branch to the helper

**Source, database-shaped** — `get_sql_source_env(conn_id, source_name)` reads a Connection and emits **one env var** `SOURCES__<SOURCE_NAME>__CREDENTIALS` containing a JSON object (`drivername`, `host`, `port`, `database`, `username`, `password`). dlt's environment provider parses a JSON object for nested values into the credentials dict — empirically verified. `source_name` must match the connector's `DLT_SOURCE_NAME` — this works because sql_database looks its credentials up **explicitly** (`dlt.secrets["sources.<DLT_SOURCE_NAME>.credentials"]` in code).

**Source, API-token-shaped** — read the token from a Connection field in the DAG and set the env var for the section dlt actually searches: for decorator-injected credentials (`dlt.secrets.value` args) the section is derived from the **module defining the `@dlt.source` function** (e.g. `SOURCES__PERSONIO__CLIENT_ID` for a source in the `personio` package) — NOT from `DLT_SOURCE_NAME` and NOT from the function name. Connector READMEs can document this wrongly; on a miss, dlt's `ConfigFieldMissingException` lists the exact keys searched — that list is ground truth.

All helpers return nothing when the Connection is absent → clean fallback to `secrets.toml`.

## Path 2: `secrets.toml` (file fallback)

The repo's `.dlt/secrets.toml` is bind-mounted to `/home/airflow/.dlt/secrets.toml`; dlt reads it automatically. Edits apply immediately (no restart). Use for local development; keep out of version control (the template's `.gitignore` handles this — root `.dlt/` is ignored, only `*_example.toml` files are committed).

## Path 3: container env vars (external databases, CI-managed)

`run_dlt_connector()` passes the worker's full environment into every connector subprocess, so any `DESTINATION__*` / `SOURCES__*` env var set at the **container level** (compose environment) reaches dlt. Flow for CI-managed credentials:

GitHub repository secrets → deploy workflow (`env:` + `envs:` passthrough in the SSH action) → persisted in the server's compose environment → all containers → every subprocess.

```bash
DESTINATION__POSTGRES__CREDENTIALS__HOST=<dwh_host>
DESTINATION__POSTGRES__CREDENTIALS__PASSWORD=<from CI secret>
SOURCES__<SOURCE_MODULE>__API_TOKEN=<from CI secret>
```

## Path 4: full `secrets.toml` in an Airflow Variable (flatten to env)

Some repos — notably Airflow 2 + airflowprovider setups and BigQuery destinations — keep an entire dlt `secrets.toml` (destination credentials, `[schema]`, `[load]`, `[runtime]`) in one Airflow Variable, conventionally `dlt_secrets_toml`. Do NOT write it to disk at runtime (races across parallel DAGs); flatten it into env vars instead — env vars are dlt's highest-precedence provider, so this reproduces the file exactly:

```python
import json

import tomllib  # stdlib >= 3.11; on 3.9/3.10: pip install tomli, import tomli as tomllib
from airflow.models import Variable
from airflow.utils.log.secrets_masker import mask_secret  # Airflow 2 path


def get_secrets_toml_env() -> dict:
    raw = Variable.get("dlt_secrets_toml")
    try:
        data = tomllib.loads(raw)
    except tomllib.TOMLDecodeError:
        # `from None`: the original message can quote a secret fragment into task logs
        raise ValueError("Variable dlt_secrets_toml is not valid TOML") from None

    env = {}

    def flatten(prefix: str, obj: dict) -> None:
        for key, value in obj.items():
            path = f"{prefix}__{key.upper()}" if prefix else key.upper()
            if isinstance(value, dict):
                flatten(path, value)
            else:
                # lists (e.g. OAuth scopes) must be JSON — dlt's env provider
                # parses JSON for complex values; str() would give a Python repr
                env[path] = json.dumps(value) if isinstance(value, list) else str(value)

    flatten("", data)
    for value in env.values():
        mask_secret(value)  # auto-masking may be off (see Guardrails)
    return env
```

`[destination.bigquery.credentials] project_id` becomes `DESTINATION__BIGQUERY__CREDENTIALS__PROJECT_ID`, etc. Merge the result into the `run_dlt_connector()` env like any other path; source credentials can still come from per-source Connections on top.

## Caveats that bite

- **Fernet key**: `AIRFLOW__CORE__FERNET_KEY` encrypts Connection passwords in the metadata DB. Set it in production (tier 2) and keep it **stable** — losing it makes all stored Connections unreadable; not setting it stores them effectively unencrypted.
- **Compose bakes CI-injected values at `up` time**: reboots and `restart: always` are fine, but a manual `docker compose up -d` on the server recreates containers **without** the CI-injected values. Container recreation must go through the deploy workflow.
- **Never pass credentials as docker `--build-arg`** — they persist readably in image layers. Runtime env injection only.
- **`docker inspect` shows env values** to anyone with docker-socket access regardless of injection path — CI injection buys central rotation, not secrecy from server admins.

## Guardrails

- NEVER commit real credentials: no `secrets.toml`, no tokens in DAG files, no real values in `*_example.toml`
- NEVER read a `secrets.toml` in full during agent sessions — targeted `grep`/`sed` only
- In production set `AIRFLOW__CORE__HIDE_SENSITIVE_VAR_CONN_FIELDS=True` (masks values in UI and logs). Flag it as a hygiene finding when a repo runs with `False` while secrets live in Variables/Connections: values render unmasked in the UI (still Fernet-encrypted at rest) **and** auto-masking of "secret"-named Variables in logs is disabled — explicit `mask_secret()` still works and must be used when reading secrets from a Variable (see Path 4)
- Keep dlt's `log_level = "WARNING"` — DEBUG can log credentials

## Quick reference

| I need to… | Do this |
|---|---|
| Point all DAGs at a different destination | Change Variable `dlt_destination` |
| Give a connector a source API token | Connection named after the connector; DAG maps `conn.password` → the connector's env var |
| Give a connector source DB credentials | Connection + `get_sql_source_env(conn_id, source_name)` |
| Provide DWH credentials from CI | `DESTINATION__*` env vars via deploy workflow → compose environment |
| Use BigQuery, or a repo keeping secrets.toml in a Variable | Path 4: flatten `dlt_secrets_toml` Variable to env vars |
| Local dev without touching the UI | Fill `.dlt/secrets.toml` from the example file |

**Related skills:** `setup-airflow3-dlt-template` · `create-airflow3-connector-dag` · `setup-airflow3-cicd` · `gemma-1password/1password` (storing/retrieving the secrets themselves)