---
name: setup-dlt-docker-operator-airflow
description: Set up an Airflow environment for DockerOperator — docker-proxy socat bridge, Docker socket mount, shared utilities, and Airflow Variables. Use when configuring Docker Compose for DockerOperator, setting up the docker-proxy service, or preparing an Airflow instance to run containerized dlt tasks.
disable-model-invocation: true
---

# Set Up Airflow for DockerOperator

Configure an Airflow Docker Compose environment to support running tasks via `DockerOperator`. This involves adding a docker-proxy service, mounting the Docker socket, and setting up the shared utilities.

## Context

`DockerOperator` requires access to a Docker daemon from inside the Airflow containers. Since mounting the Docker socket directly into the Airflow containers can cause permission issues, the recommended approach is to use a **socat proxy** (`alpine/socat`) that listens on a TCP port and forwards to the Docker socket. Airflow tasks then connect to `tcp://docker-proxy:2375` instead of the socket directly.

This setup applies to both local development (Docker Compose) and production (VM-based) environments.

## Steps

### Docker Compose Setup (Local Development)

1. Add the docker-proxy service to `docker-compose.yaml`.

   ```yaml
   services:
     # ... existing Airflow services ...

     # For DockerOperator - TCP proxy to Docker socket
     docker-proxy:
       image: alpine/socat
       command: "TCP4-LISTEN:2375,fork,reuseaddr UNIX-CONNECT:/var/run/docker.sock"
       ports:
         - "127.0.0.1:2376:2375"  # Only listen on localhost
       volumes:
         - /var/run/docker.sock:/var/run/docker.sock
   ```

2. Mount the Docker socket in the Airflow common configuration (the `x-airflow-common` anchor).

   ```yaml
   x-airflow-common:
     &airflow-common
     # ... existing config ...
     volumes:
       - ${AIRFLOW_PROJ_DIR:-.}/dags:/opt/airflow/dags
       - ${AIRFLOW_PROJ_DIR:-.}/logs:/opt/airflow/logs
       - ${AIRFLOW_PROJ_DIR:-.}/config:/opt/airflow/config
       - ${AIRFLOW_PROJ_DIR:-.}/plugins:/opt/airflow/plugins
       # For DockerOperator - mount Docker socket
       - /var/run/docker.sock:/var/run/docker.sock
   ```

3. Create the shared utilities directory and files.

   ```bash
   mkdir -p dags/utils
   ```

   Copy the utilities from the **`create-dlt-docker-operator-dag`** skill's `assets/` directory:
   - `assets/utils_common.py` → `dags/utils/common.py` — provides `get_dlt_destination()`, `get_dlt_secrets_toml_base64()`, `sanitize_task_id()`
   - `assets/utils_dbt.py` → `dags/utils/dbt.py` — provides `create_dbt_task()` (optional, for dbt workloads)

   > **Note:** These utility files live in the companion skill `create-dlt-docker-operator-dag`, not in this skill's directory. If using the uv approach instead, see `setup-dlt-uv-connector-airflow` which has its own `assets/common.py` with a different implementation.

4. Install the Docker provider package in the Airflow environment. Either add to `_PIP_ADDITIONAL_REQUIREMENTS` for quick testing:

   ```yaml
   _PIP_ADDITIONAL_REQUIREMENTS: apache-airflow-providers-docker
   ```

   Or build a custom Airflow image with the provider pre-installed (recommended for production).

5. Configure Airflow Variables via the UI or CLI:

   | Variable | Value | Description |
   |----------|-------|-------------|
   | `dlt_destination` | `postgres` | Target destination for dlt connectors |
   | `dlt_secrets_toml` | Full TOML content | The contents of `secrets.toml` (stored as a string) |

6. Set environment variables in `.env` for the Airflow project.

   ```env
   AIRFLOW_PROJ_DIR=/path/to/airflow/project
   AIRFLOW_UID=1000
   AIRFLOW_GID=0
   AIRFLOW__CORE__FERNET_KEY='<generate-a-fernet-key>'
   ```

7. Restart Airflow to pick up the changes.

   ```bash
   docker compose down && docker compose up -d
   ```

### Production Setup (VM-Based)

8. On a production VM where Airflow runs directly (not in Docker Compose), configure the docker-proxy as a systemd service or use Docker's TCP socket directly.

   For the `docker run` production pattern, pass the Docker socket and proxy URL:

   ```bash
   docker run --rm \
     -v /var/run/docker.sock:/var/run/docker.sock \
     --env DLT_DESTINATION=postgres \
     --env DLT_SOURCE_NAME=<connector_name> \
     --env DLT_SECRETS_TOML_BASE64=$(base64 -w 0 /path/to/secrets.toml) \
     dlt-connector-<connector_name>:latest
   ```

   In production DAGs, use `docker_url="unix://var/run/docker.sock"` if the Airflow process has direct socket access, or `tcp://docker-proxy:2375` if using the proxy.

## Validation

- [ ] `docker-proxy` service is running: `docker ps | grep docker-proxy`
- [ ] Airflow can see DAGs: check the UI for import errors
- [ ] A test DAG with DockerOperator runs successfully
- [ ] Airflow Variables are configured (`dlt_destination`, `dlt_secrets_toml`)

## Examples

**Full docker-compose snippet** (see `assets/docker-compose-snippet.yaml`):

```yaml
# Add to x-airflow-common volumes:
- /var/run/docker.sock:/var/run/docker.sock

# Add as a new service:
docker-proxy:
  image: alpine/socat
  command: "TCP4-LISTEN:2375,fork,reuseaddr UNIX-CONNECT:/var/run/docker.sock"
  ports:
    - "127.0.0.1:2376:2375"
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
```

**Adapting an EWAH Airflow setup:**

If migrating from EWAH (which runs connectors inside the Airflow container), the key changes are:

1. Add the docker-proxy service to `docker-compose.yml`
2. Mount the Docker socket in Airflow containers
3. Install `apache-airflow-providers-docker`
4. Replace EWAH DAGs with DockerOperator DAGs (see `create-dlt-docker-operator-dag` skill)
5. Store dlt secrets in an Airflow Variable instead of Airflow Connections
6. Build and tag dlt connector Docker images on the machine