---
name: setup-airflow3-dlt-template
description: Stand up the self-contained Airflow 3 template for running dlt connectors as subprocess tasks. Use when setting up a new Airflow 3 environment from the airflow3-best-practice template, bootstrapping the compose stack, or configuring destination Connections and Variables for dlt connectors.
disable-model-invocation: true
---

# Set Up the Airflow 3 dlt Template

Stand up a new Airflow 3 environment from the [airflow3-best-practice](https://github.com/Gemma-Analytics/airflow3-best-practice) template — a self-contained compose stack that runs dlt connectors as `uv` subprocess tasks with bundled, runnable example connectors.

## Context

This is the current-generation pattern: Airflow 3, `@task` DAGs, `dags/utils/dlt.py` utilities, credentials from Airflow Connections (with `secrets.toml` fallback), connectors as hard copies bind-mounted for live editing. The template ships three example connectors (`pokemon`, `fx`, `sql_database`) and their DAGs, chosen to demonstrate the common cases.

For **existing** Airflow 2.x / EWAH / airflowprovider repos, use `setup-dlt-uv-connector-airflow` instead — its modern hard-copy variant ports this template's building blocks (`utils/dlt.py`, subprocess isolation, mounted `connectors/`) into the existing repo without a full template migration; its legacy variant covers lockfile-based setups.

## Prerequisites

- Docker Engine + compose plugin
- Git access to `github.com/Gemma-Analytics/airflow3-best-practice`
- A destination database reachable from the containers — or none: set Variable `dlt_destination=duckdb` for the zero-setup smoke-test mode (no credentials, writes a gitignored `.duckdb` file per run), or start a local dev Postgres (see Examples)

## Steps

### 1. Bootstrap the new repository

The deciding question is **who can read the new repo**:

**Client-owned, external, or separate-account repos — clean history is mandatory.** The template's content travels, its internal commit history never does. Two equivalent paths:

- Target account has template access: GitHub's "Use this template" (fresh repo, single initial commit, no fork relationship), then clone the new repo.
- No template access on the target side: clone, strip the history, start fresh:

```bash
git clone --depth 1 git@github.com:Gemma-Analytics/airflow3-best-practice.git <target_dir>
cd <target_dir> && rm -rf .git
git init -b main
git add -A && git commit -m "chore: bootstrap from airflow3-best-practice template"
git remote add origin git@github.com:<owner>/<new_repo>.git
git push -u origin main
```

**Gemma-internal repos — keeping history is fine.** A plain clone with a re-pointed remote is acceptable (and keeps the template lineage traceable):

```bash
git clone git@github.com:Gemma-Analytics/airflow3-best-practice.git <target_dir>
cd <target_dir>
git remote set-url origin git@github.com:Gemma-Analytics/<new_repo>.git
git push -u origin main
```

In **both** cases: do not fork (permanent upstream tie). The new repo is deliberately independent — future template-level improvements are adopted manually; connector updates have their own path via `update-airflow3-connectors`. When the machine hosts multiple GitHub identities, make sure the push goes out as the right one (SSH host alias or `gh auth switch`).

### 2. Bootstrap the environment file

Only non-secret bootstrap values belong here (see the `airflow3-dlt-credentials` skill for the full credential model):

```bash
mkdir -p ./dags ./logs ./plugins
echo -e "AIRFLOW_UID=$(id -u)" >> .env
echo -e "AIRFLOW_PROJ_DIR=$(pwd)" >> .env
# local development (credentials visible in UI):
echo -e "AIRFLOW__CORE__HIDE_SENSITIVE_VAR_CONN_FIELDS=False" >> .env
# production: set the flag to True instead
```

### 3. Build and start

```bash
docker compose build   # image adds uv + docker/fab providers
docker compose up -d
```

UI at `http://localhost:8080` (default `airflow` / `airflow`).

### 4. Configure destination credentials

In the UI (Admin → Connections + Variables) — the UI takes precedence over the mounted `.dlt/secrets.toml` fallback:

**Variable:** `dlt_destination` = `postgres`, `snowflake`, or `duckdb` (smoke-test mode — skip the Connections below entirely)

**Connection `dlt_postgres`** (Connection Type: Postgres): Host, Schema = database name, Login, Password, Port.

**Connection `dlt_snowflake`** (Connection Type: Generic): Host = account identifier, Schema = database, Login, Password (or leave empty for key auth), Extra = `{"warehouse": "<wh>", "role": "<role>", "private_key": "...", "private_key_passphrase": "..."}`.

### 5. Verify with the bundled examples

Trigger `extract_load_pokemon` first — it needs **no source credentials** (public API), so a success proves the whole chain: mount → uv subprocess → destination credentials → load.

```bash
docker compose exec airflow-scheduler airflow dags list-import-errors
docker compose exec airflow-scheduler airflow dags trigger extract_load_pokemon
```

Then optionally `extract_load_fx` (env-var parameterized endpoints) and `extract_load_sql_database` (source credentials from a Connection — create Connection `airflow_metadata_db` per its DAG docstring; it reads Airflow's own metadata DB so it works out of the box).

### 6. Adapt to the client

- The bundled example connectors/DAGs are removable — they are teaching material, not obligations
- Add real connectors with the `update-airflow3-connectors` skill (selection + copy) and `create-airflow3-connector-dag` (DAG)
- Wire CI/CD with the `setup-airflow3-cicd` skill

## Validation

- [ ] `airflow dags list-import-errors` returns no errors
- [ ] `extract_load_pokemon` runs to success; task logs show dlt load info
- [ ] Live-edit check: change a line in `connectors/pokemon/pokemon_pipeline.py` on the host, re-trigger **without restarting anything** — the change appears in the task log (validates the bind mount)
- [ ] Read the template's `CLAUDE.md` — it is the agent-facing reference for utils, conventions, and the temp-file lifecycle

## Examples

**Local dev Postgres as destination:**

```bash
docker run -d --name postgres_dwh \
  -e POSTGRES_USER=<user> -e POSTGRES_PASSWORD=<password> \
  -p 5434:5432 postgres:latest
# Use host.docker.internal (not localhost) as Connection host from inside compose
```

**Key template facts** (details in the template's `CLAUDE.md`/`README.md`):

| Aspect | Behavior |
|---|---|
| Connector runs | `uv run --no-dev python <name>_pipeline.py`, isolated venv per connector in `/tmp/dlt_venvs/<name>` (cached) |
| Connector edits | Apply on the next DAG run — no rebuild, no restart |
| Pipeline working dirs | `/tmp/dlt_pipelines/<source>`, auto-deleted after successful runs (state restored from destination); kept on failure for retry resume |
| Credential precedence | UI Connection > container env vars > `secrets.toml` |

**Related skills:** `update-airflow3-connectors` · `create-airflow3-connector-dag` · `airflow3-dlt-credentials` · `setup-airflow3-cicd` · `setup-dlt-uv-connector-airflow` (Airflow 2/EWAH generation)