# CI/CD Deployment Reference — sync-dlt-connectors

Detailed CI/CD integration guides for syncing dlt connectors in production deployments.

## How production deployment works

1. CI runner checks out the Airflow repo
2. CI runner runs `./sync-connectors.sh` (fetches pinned connector versions via git)
3. CI runner builds the Docker image with `COPY connectors /opt/airflow/connectors`
4. Image is pushed to a registry and deployed

## GitHub Actions — complete workflow

Create `.github/workflows/deploy.yml` in the Airflow repository.

```yaml
name: Deploy Airflow

on:
  push:
    branches: [main]
  workflow_dispatch:       # Allow manual triggers

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # SSH access to the connectors monorepo (required by sync-connectors.sh)
      # Supports passphrase-protected deploy keys via ssh-agent
      - name: Configure SSH for connectors repo
        env:
          DEPLOY_KEY: ${{ secrets.CONNECTORS_DEPLOY_KEY }}
          DEPLOY_KEY_PASSPHRASE: ${{ secrets.CONNECTORS_DEPLOY_KEY_PASSPHRASE }}
        run: |
          mkdir -p ~/.ssh
          ssh-keyscan github.com >> ~/.ssh/known_hosts
          echo "$DEPLOY_KEY" > /tmp/deploy_key
          chmod 600 /tmp/deploy_key
          printf '#!/bin/sh\necho "%s"\n' "$DEPLOY_KEY_PASSPHRASE" > /tmp/askpass.sh
          chmod +x /tmp/askpass.sh
          eval "$(ssh-agent -s)"
          SSH_ASKPASS=/tmp/askpass.sh SSH_ASKPASS_REQUIRE=force ssh-add /tmp/deploy_key
          rm -f /tmp/deploy_key /tmp/askpass.sh
          echo "SSH_AUTH_SOCK=$SSH_AUTH_SOCK" >> "$GITHUB_ENV"
          echo "SSH_AGENT_PID=$SSH_AGENT_PID" >> "$GITHUB_ENV"

      - name: Sync dlt connectors
        run: ./sync-connectors.sh

      - name: Build production image
        run: |
          docker build \
            --target prod_build \
            --build-arg arg_airflow__core__fernet_key=${{ secrets.AIRFLOW_FERNET_KEY }} \
            --build-arg arg_airflow__core__sql_alchemy_conn=${{ secrets.AIRFLOW_SQL_ALCHEMY_CONN }} \
            --build-arg arg_airflow__smtp__smtp_password=${{ secrets.AIRFLOW_SMTP_PASSWORD }} \
            --build-arg arg_airflow__webserver__secret_key=${{ secrets.AIRFLOW_WEBSERVER_SECRET_KEY }} \
            -t airflow-prod:${{ github.sha }} \
            .

      # Example: push to GitHub Container Registry
      # - name: Log in to GHCR
      #   run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io -u ${{ github.actor }} --password-stdin
      # - name: Push image
      #   run: |
      #     docker tag airflow-prod:${{ github.sha }} ghcr.io/${{ github.repository }}:${{ github.sha }}
      #     docker push ghcr.io/${{ github.repository }}:${{ github.sha }}

      # Example: deploy to a server
      # - name: Deploy
      #   run: |
      #     ssh ${{ secrets.DEPLOY_HOST }} "docker pull ghcr.io/${{ github.repository }}:${{ github.sha }} && docker compose up -d"
```

## Deploy key setup

```bash
# Generate a deploy key pair (with optional passphrase for extra security)
ssh-keygen -t ed25519 -C "ci-deploy-key" -f deploy_key

# Add the PUBLIC key as a read-only deploy key in the connectors monorepo:
# GitHub -> dlt-connectors -> Settings -> Deploy keys -> Add deploy key

# Add the PRIVATE key as a repository secret in the Airflow repo:
# GitHub -> Airflow repo -> Settings -> Secrets -> CONNECTORS_DEPLOY_KEY

# If you set a passphrase, also add it as a secret:
# GitHub -> Airflow repo -> Settings -> Secrets -> CONNECTORS_DEPLOY_KEY_PASSPHRASE
```

## GitLab CI — complete pipeline

If the client uses GitLab, create `.gitlab-ci.yml` in the Airflow repository.

```yaml
stages:
  - build
  - deploy

variables:
  DOCKER_TLS_CERTDIR: "/certs"

build:
  stage: build
  image: docker:latest
  services:
    - docker:dind
  before_script:
    - apk add --no-cache git bash openssh-client
  script:
    - ./sync-connectors.sh
    - docker build --target prod_build -t airflow-prod:$CI_COMMIT_SHA .
    # Push to GitLab Container Registry
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker tag airflow-prod:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker tag airflow-prod:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:latest
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker push $CI_REGISTRY_IMAGE:latest

deploy:
  stage: deploy
  script:
    - echo "Deploy to production server..."
    # ssh deploy@$DEPLOY_HOST "docker compose pull && docker compose up -d"
  only:
    - main
```

GitLab CI runners have automatic access to repositories in the same GitLab group, so no extra SSH setup is needed if the connectors repo fork lives in the same group.

## Dockerfile — baking connectors into the production image

Ensure the Dockerfile's production stage includes a `COPY connectors` step.

```dockerfile
# Production stage
FROM dev_build as prod_build

# Bake DAGs and dlt connectors into image
RUN rm -rf /opt/airflow/dags
COPY dags /opt/airflow/dags
COPY connectors /opt/airflow/connectors
```

In production, do NOT volume-mount `./connectors`. The image contains everything it needs. Volume mounts are only used in local development (via `docker-compose.yml`).