---
name: setup-dlt-uv-connector-airflow
description: Set up an Airflow 2 environment to run dlt connectors via uv with dependency isolation. Use when bringing standalone dlt connectors into any Airflow 2 repo — including EWAH or airflowprovider repos with no connectors/ folder (hybrid variant reusing the Airflow 3 template utilities) — or when maintaining the legacy lockfile distribution (sync-connectors.sh, common.py, dlt_secrets_toml Variable).
disable-model-invocation: true
---

# Set Up Airflow for uv-Based dlt Connectors

Configure an Airflow environment to run dlt connectors using `uv` for dependency isolation. Each connector runs in its own virtual environment managed by uv, eliminating the need for Docker-in-Docker, docker-proxy services, or a container registry.

## Context

Instead of running connectors inside Docker containers via `DockerOperator`, this approach runs them directly on the Airflow host using `uv`. Each connector is a self-contained Python project with its own `pyproject.toml` and `uv.lock`. When `uv run` is invoked, it creates/uses an isolated virtual environment per connector — dependencies never interfere with each other or with Airflow itself.

Subprocess isolation is what makes this work at all: in-process frameworks (EWAH, airflowprovider) pin an old dlt (e.g. `dlt==0.5.4`) in the Airflow interpreter, while the monorepo connectors require `dlt>=1.20` — they cannot share one interpreter. Nothing here needs Airflow 3: there is no `@task.subprocess` decorator in any Airflow version; a plain `PythonOperator` / `@task` spawning the subprocess works on Airflow 2.

### Choose a variant

| Variant | When |
|---|---|
| **Modern hard-copy** (below, recommended) | New work on any Airflow 2 repo — including EWAH/airflowprovider repos with **no** `connectors/` folder yet. Reuses the Airflow 3 template's `dags/utils/dlt.py` runner, which is Airflow-2-compatible |
| **Legacy lockfile** | Repos already running `connectors.lock.yml` + `sync-connectors.sh` + `dags/utils/common.py`. Keep maintaining as documented; migrate to the modern variant opportunistically |

In the legacy variant, connectors are distributed via a lockfile (`connectors.lock.yml`) and a sync script (`sync-connectors.sh`) with per-connector version pinning, and `connectors/` is gitignored. In the modern variant, connectors are **committed hard copies** updated with the diff-aware flow of the `update-airflow3-connectors` skill.

## Prerequisites

- An existing Airflow setup (EWAH-based, airflowprovider-based, or standard)
- Git access to the dlt-connectors monorepo
- Connectors already developed in the monorepo (tagged versions required only for the legacy variant)

## Steps — modern hard-copy variant

Port standalone connectors into an existing Airflow 2 repo by reusing the Airflow 3 template's building blocks. Battle-tested on Airflow 2.11 / Python 3.12 alongside an in-process airflowprovider framework.

### 1. Install uv in the image

Same as legacy step 1 below (`RUN pip install uv` in the Dockerfile). If deploys split into pull-only vs rebuild workflows, note the Dockerfile change auto-fires the rebuild workflow on merge — no manual rebuild step needed.

### 2. Hard-copy connectors from the monorepo

Create `connectors/` and copy connectors from dlt-connectors **main** — use the `update-airflow3-connectors` skill (its `update_check.sh` handles the clone, diff and new-connector listing; its post-copy checks apply here too):

- Strip artifacts: `.venv/`, `__pycache__/`, `*.duckdb`, real credential TOMLs (copy `.dlt/*_example.toml` only)
- Verify the standard env-var pattern (`DLT_DESTINATION`, `DLT_SOURCE_NAME`, `DLT_DATASET_NAME`, `DLT_PIPELINES_DIR`) is present in `<name>_pipeline.py` — **add it if missing**. Without `DLT_PIPELINES_DIR` support, all runs share `~/.dlt/pipelines/<pipeline_name>`, and two parallel tasks with the same pipeline name corrupt dlt local state. Only if the connector must stay unmodified: serialize such tasks instead
- `connectors/` is committed in this variant — do NOT gitignore it

### 3. Copy the runner utilities

Copy `dags/utils/dlt.py` from [airflow3-best-practice](https://github.com/Gemma-Analytics/airflow3-best-practice) — `run_dlt_connector`, `_cleanup_pipelines_dir`, `get_destination_env`, `get_sql_source_env`, `sanitize_task_id` run on Airflow 2.11 unmodified (the Airflow import is lazy and Airflow-2-compatible). Per-connector venvs land in `/tmp/dlt_venvs/<name>` — deliberately **off** any bind mount (avoids host-path/uid/shadowing issues).

The repo-specific part is **credential sourcing**:

- Destination Connections via `get_destination_env()` cover postgres/snowflake/duckdb only — no BigQuery branch
- Repos that keep a full dlt `secrets.toml` in an Airflow Variable (common with airflowprovider/BigQuery): use the flatten-to-env pattern — see the `airflow3-dlt-credentials` skill, "secrets-toml Variable" path, including its masking/exception hardening

### 4. Mount vs bake

**Mount `connectors/`** (like `dags/`) when deploys are git-pull-based: connector edits deploy via cheap `git pull`, no image rebuild. Add the mount **everywhere containers are defined** — `docker-compose.yml` AND any raw `docker run` blocks in deploy workflows — and put `connectors/**` on the pull-only deploy track. **Bake** (see the production section of the legacy variant) when workers are ephemeral or the server has no git checkout.

### 5. Create DAGs

The DAG patterns from the `create-airflow3-connector-dag` skill work on Airflow 2.11 with one adjustment: swap the Airflow 3 imports (`from airflow.sdk import Variable, dag, task`) for the Airflow 2 equivalents (`from airflow.decorators import dag, task` / `from airflow.models import Variable`). Always pass a unique `DLT_PIPELINES_DIR` per task.

### Validation (modern variant)

- [ ] `uv` available in the container; connector venv builds on first run under `/tmp/dlt_venvs/<name>`
- [ ] A DAG run loads to the destination; task log shows dlt row counts
- [ ] Parallel tasks use distinct `DLT_PIPELINES_DIR` values (or are serialized)
- [ ] `git status` shows no real credential files staged

## Steps — legacy lockfile variant

### Install uv in the Airflow image

1. Add `uv` to the Airflow Dockerfile.

   For EWAH-based setups:

   ```dockerfile
   FROM gemmaanalytics/ewah:<version> as dev_build

   # For dlt connectors (run via uv with isolated Python)
   RUN pip install uv
   ```

   For standard Apache Airflow setups:

   ```dockerfile
   FROM apache/airflow:<version>

   RUN pip install uv
   ```

   `uv` will automatically download the required Python version (e.g., 3.12+) when running connectors, independent of the system Python in the Airflow image.

### Set up connector distribution

2. Copy `sync-connectors.sh` and `contribute-connector.sh` from the dlt-connectors monorepo `dist/` directory to the Airflow repo root.

   ```bash
   cp /path/to/dlt-connectors/dist/sync-connectors.sh ./
   cp /path/to/dlt-connectors/dist/contribute-connector.sh ./
   chmod +x sync-connectors.sh contribute-connector.sh
   ```

3. Create `connectors.lock.yml` listing the connectors this client needs.

   ```yaml
   repo: git@github.com:Gemma-Analytics/dlt-connectors.git

   connectors:
     google_ads:
       version: google_ads/v0.1.0
     # Add more connectors as needed
   ```

4. Add `connectors/` to `.gitignore` — the lockfile is the source of truth.

   ```
   # dlt connectors (fetched by sync-connectors.sh)
   connectors/
   ```

5. Run the sync script to fetch connectors.

   ```bash
   ./sync-connectors.sh
   ```

   The sync script automatically sets world-writable permissions (`chmod -R a+w`) on each connector directory. This is required because the Airflow user inside Docker needs write access to create `.dlt/secrets.toml` (written at runtime by the utility) and `.venv/` (created by `uv` on first run).

   **Important: sync is one-way (monorepo → client repo).** Running `sync-connectors.sh` overwrites the local `connectors/<name>/` directory with the version pinned in `connectors.lock.yml`. Any local edits to connector code will be lost on the next sync. If you need to fix a bug or add a feature to a connector, use `contribute-connector.sh` to submit the change back to the monorepo as a PR. Once merged and auto-tagged, update `connectors.lock.yml` to the new version and re-sync. See the `contribute-connector` skill for the full workflow.

### Mount connectors in Airflow

6. Add the connectors volume mount to `docker-compose.yml` for the scheduler and webserver services.

   ```yaml
   services:
     webserver:
       volumes:
         - ./dags:/opt/airflow/dags
         - ./connectors:/opt/airflow/connectors

     scheduler:
       volumes:
         - ./dags:/opt/airflow/dags
         - ./connectors:/opt/airflow/connectors
   ```

7. If you had a `docker-proxy` service for DockerOperator, it can be removed.

### Configure Airflow Variables

8. Set up Airflow Variables for dlt connector configuration.

   | Variable | Value | Description |
   |----------|-------|-------------|
   | `dlt_destination` | `postgres` | Target destination for dlt connectors |
   | `dlt_secrets_toml` | Raw TOML content | The full contents of `secrets.toml` (written to disk at runtime by the utility) |

   Set via CLI:

   ```bash
   docker exec <scheduler_container> airflow variables set dlt_destination postgres
   docker exec <scheduler_container> airflow variables set dlt_secrets_toml "$(cat /path/to/secrets.toml)"
   ```

### Create shared utilities

9. Create the shared utility module at `dags/utils/common.py`. Make sure `dags/utils/__init__.py` exists too.

   Copy the template from `${CLAUDE_SKILL_DIR}/assets/common.py` — it provides:
   - `get_dlt_destination()` — reads the destination from Airflow Variable
   - `get_dlt_secrets_toml()` — reads and validates secrets TOML from Airflow Variable
   - `run_dlt_connector(connector_name, source_name, extra_env=None)` — writes secrets.toml, then runs the pipeline via `uv run` in a subprocess

   See the `create-dlt-uv-connector-dag` skill for how to use this in DAGs.

### Rebuild and restart

10. Rebuild the Airflow image and restart services.

    ```bash
    docker compose build
    docker compose down && docker compose up -d
    ```

### Production deployment — bake connectors into the Docker image

In production, connectors should be **baked into the Docker image** at build time rather than volume-mounted. This means:

- No `sync-connectors.sh` on the production server
- No git access from the production server
- No volume mounts for connectors
- Connectors are versioned with the image — deterministic deployments

11. Add a `COPY connectors` step to the **production stage** of the Dockerfile.

    For EWAH-based setups (multi-stage build):

    ```dockerfile
    ###########################################################
    ## Multi-stage build: Second stage is used in production ##
    ###########################################################

    FROM dev_build as prod_build

    # ... production ENV vars ...

    # Bake DAGs and dlt connectors into image
    RUN rm -rf /opt/airflow/dags
    COPY dags /opt/airflow/dags
    COPY connectors /opt/airflow/connectors
    ```

    For standard Airflow setups:

    ```dockerfile
    FROM apache/airflow:<version>

    RUN pip install uv

    COPY dags /opt/airflow/dags
    COPY connectors /opt/airflow/connectors
    ```

    Inside the image, connectors are writable by default (no volume mount permission issues). The `run_dlt_connector()` utility writes `.dlt/secrets.toml` and `uv run` creates `.venv/` at runtime.

12. Set up CI/CD to sync connectors before building the image. The CI/CD runner fetches connectors from the monorepo (where git access is available), then builds the image with them baked in. See `${CLAUDE_SKILL_DIR}/references/cicd-deployment-reference.md` for complete GitHub Actions and GitLab CI workflow templates, deploy key setup, and SSH configuration.

13. In production, run the image **without** volume mounts for connectors.

    ```bash
    # Production — connectors are baked into the image
    docker run -d --name airflow-scheduler \
      airflow-prod:latest scheduler

    # Do NOT mount ./connectors in production
    ```

    **Local development** still uses volume mounts (from `docker-compose.yml`) so you can iterate on DAGs and test connector changes without rebuilding.

## Validation

- [ ] `uv` is available inside the Airflow container: `docker exec <scheduler> uv --version`
- [ ] Connectors synced locally: `ls connectors/` shows expected connectors
- [ ] Connector runs inside container: `docker exec <scheduler> bash -c "cd /opt/airflow/connectors/<name> && uv run python -c 'import dlt; print(\"OK\")'"` 
- [ ] Airflow Variables configured (`dlt_destination`, `dlt_secrets_toml`)
- [ ] DAGs load without import errors in Airflow UI
- [ ] CI/CD pipeline syncs connectors and builds image successfully

## Examples

**Updating a connector version:**

```bash
# Edit connectors.lock.yml to bump version
# e.g., google_ads/v0.1.0 → google_ads/v0.2.0

./sync-connectors.sh google_ads   # Sync only the updated connector

# Restart Airflow to pick up changes
docker compose restart scheduler
```

**Adding a new connector:**

```bash
# 1. Add to connectors.lock.yml
# 2. Sync
./sync-connectors.sh
# 3. Create a DAG (see create-dlt-uv-connector-dag skill)
# 4. Restart
docker compose restart scheduler
```

**EWAH compatibility:** For the complete EWAH setup checklist and important compatibility notes (Python version constraints, Airflow version limitations), see `${CLAUDE_SKILL_DIR}/references/ewah-compatibility-reference.md`.

**Related skills:** `update-airflow3-connectors` (hard-copy diff/add mechanics) · `create-airflow3-connector-dag` (DAG patterns, Airflow-2-portable) · `airflow3-dlt-credentials` (credential paths incl. the secrets-toml Variable flatten) · `setup-airflow3-dlt-template` (greenfield Airflow 3 instead of retrofitting Airflow 2)