---
name: setup-airflow3-cicd
description: Wire GitHub Actions CI and deployment workflows for an Airflow 3 template repository — PR checks plus a pull-only sync vs full-redeploy split. Use when setting up CI/CD for an Airflow 3 dlt repo, provisioning the deployment server, configuring deploy secrets, or deciding between restart-free syncs and full rebuilds.
disable-model-invocation: true
---

# Set Up Airflow 3 CI/CD

Wire the CI and deployment workflows for an Airflow 3 template repository, using the [airflow3-best-practice](https://github.com/Gemma-Analytics/airflow3-best-practice) workflows as the source.

## Context

The template's deployment design exploits its runtime properties: `dags/` hot-reloads via the dag-processor, and `connectors/` runs as a fresh `uv` subprocess per task from a bind mount — so merges touching only those paths need a **`git pull` on the server and nothing else**. Everything else (Dockerfile, compose, utils) needs a rebuild. Three workflows encode this:

| Workflow | Trigger | Action |
|---|---|---|
| `ci.yml` | every PR | black check, Python syntax check, `uv lock --check` per connector, `docker compose config` validation — no secrets needed |
| `sync_dags_connectors.yml` | merge touching only `dags/`, `connectors/`, `**.md` | SSH → `git pull` (no rebuild, no restart) |
| `deploy_airflow.yml` | merge touching anything else | SSH → `git pull` + `docker compose build` + `up -d --remove-orphans` + dangling-image prune |

Both deployment workflows share a `concurrency: airflow-deploy` group so their SSH sessions never race.

## Prerequisites

- An Airflow 3 template repository on GitHub (see `setup-airflow3-dlt-template`)
- A deployment server (VM) reachable via SSH
- Repository admin access (to add secrets)

## Steps

### 1. Copy the workflows

Copy `.github/workflows/{ci.yml,sync_dags_connectors.yml,deploy_airflow.yml}` from the template repo. If the repo was created from the template, they are already present.

### 2. Provision the server (one-time, manual)

The workflows only `git pull` and run compose — everything else is provisioned by hand:

1. Install Docker Engine + compose plugin; create a deploy user with docker group membership
2. Give the server read access to the repository (deploy key or machine-user SSH key) and clone it:
   ```bash
   sudo git clone git@github.com:<org>/<repo>.git /opt/<repo>
   ```
3. Create the environment file in the checkout with the non-secret bootstrap values (`AIRFLOW_UID`, `AIRFLOW_PROJ_DIR`, `AIRFLOW__CORE__HIDE_SENSITIVE_VAR_CONN_FIELDS=True`)
4. Provide data credentials (Airflow UI Connections after first start, or `.dlt/secrets.toml`) — see the `airflow3-dlt-credentials` skill
5. First start: `docker compose build && docker compose up -d`; create the admin user; configure Connections
6. **The checkout path must match the `cd` in both workflow scripts** — adjust them together if you deviate from `/opt/<repo>`

### 3. Add GitHub repository secrets

| Secret | Value |
|---|---|
| `AIRFLOW_HOST` | server hostname/IP |
| `AIRFLOW_USERNAME` | deploy user from step 2.1 |
| `AIRFLOW_SSH_KEY` | private key for that user |

### 4. Pass infrastructure secrets through the deploy workflow

Infrastructure values Airflow needs at boot (tier 2 of the credential model) are injected from GitHub secrets at deploy time — data credentials do NOT go here (they belong in Connections):

```yaml
      - name: Deploy via SSH
        uses: appleboy/ssh-action@v1
        env:
          AIRFLOW__CORE__FERNET_KEY: ${{ secrets.AIRFLOW_FERNET_KEY }}
          AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: ${{ secrets.AIRFLOW_METADATA_DB_CONN }}
        with:
          envs: AIRFLOW__CORE__FERNET_KEY,AIRFLOW__DATABASE__SQL_ALCHEMY_CONN
          ...
```

Ensure `docker-compose.yaml` interpolates these (e.g. `AIRFLOW__CORE__FERNET_KEY: ${AIRFLOW__CORE__FERNET_KEY:-}` in the shared environment block).

### 5. Preserve the design decisions when adapting

- **No `compose down` in deploys** — `up -d` after `build` recreates only changed containers; a `down` takes the whole stack offline for the build duration. A commented-out `down` exists for deliberate clean-slate deploys.
- **`docker image prune -f` only** (dangling images from rebuilds). **Never add `volume prune`** — `postgres-db-volume` holds the Airflow metadata DB (Connections, Variables, run history).
- **Keep the shared concurrency group** on both deployment workflows.
- **Compose bakes CI-injected env at `up` time** — container recreation must go through the deploy workflow, or a manual `compose up` drops the injected values.
- **Repos deploying via raw `docker run` instead of compose** (some Airflow 2 setups): the same split applies, but bind mounts must be added to **every** `docker run` block in the workflows, not just a compose file. Keep `connectors/**` and `dags/**` on the pull-only track; a Dockerfile change (e.g. adding uv) lands on the rebuild track automatically via the path filter — no manual rebuild step.

### 6. Commit and open a PR

Never commit to `main` — work on a feature branch. After configuring, **confirm with the user**, then:

```bash
git checkout -b feat/cicd-workflows   # skip if already on one
git add .github/
git commit -m "feat: add CI and deployment workflows"
git push -u origin feat/cicd-workflows
gh pr create --fill
```

The PR itself doubles as the first CI validation run (next section).

## Validation

- [ ] Open a test PR → `ci.yml` runs green (no secrets required)
- [ ] Merge a change touching only `dags/` → `sync_dags_connectors.yml` fires, `deploy_airflow.yml` does not; server shows the new commit without container restarts
- [ ] Merge a change touching `docker-compose.yaml` → full deploy fires; stack recreated with new config
- [ ] `docker volume ls` on the server still shows `postgres-db-volume` after a deploy

## Troubleshooting

| Issue | Solution |
|---|---|
| Both workflows fire on one merge | Expected when a push touches both path sets — the concurrency group serializes them and `git pull` is idempotent |
| Deploy succeeds but UI Connections gone | Metadata volume was pruned or Fernet key changed — restore key; never volume-prune |
| Sync fired but DAG unchanged in UI | dag-processor refresh interval; wait or check `airflow dags list-import-errors` |
| `git pull` fails on the server | Deploy key lost or checkout path diverged from the workflows' `cd` |

**Related skills:** `setup-airflow3-dlt-template` · `airflow3-dlt-credentials` · `gemma-deployment-security/audit-cicd-workflows` (audit the result)