---
name: sync-dlt-connectors
description: Set up and manage dlt connector distribution from the central monorepo to client Airflow repos. Use when syncing connectors, pinning versions in connectors.lock.yml, running sync-connectors.sh, updating connector versions, or setting up sparse checkout distribution to a client repo.
---

# Sync dlt Connectors

Set up the distribution mechanism to fetch dlt connectors from the central monorepo into a client Airflow repository. Each connector is pinned to a specific version (git tag) and fetched independently.

## Context

The dlt-connectors monorepo contains all connectors. Client Airflow repos consume only the connectors they need, pinned to specific versions via `connectors.lock.yml`. A `sync-connectors.sh` script handles fetching the right version of each connector's source code.

Connector versions use per-connector git tags following the pattern `<connector_name>/v<semver>` (e.g., `google_ads/v1.0.0`). This allows independent versioning — updating one connector doesn't affect others.

## Prerequisites

- Git access to the dlt-connectors monorepo (SSH or HTTPS)
- Connectors tagged in the monorepo (e.g., `google_ads/v0.1.0`)
- `uv` installed on the machine running the sync (local dev or CI runner)

## Steps

### Initial setup

1. Copy the distribution scripts from the dlt-connectors monorepo.

   ```bash
   cp /path/to/dlt-connectors/dist/sync-connectors.sh ./
   cp /path/to/dlt-connectors/dist/contribute-connector.sh ./
   chmod +x sync-connectors.sh contribute-connector.sh
   ```

2. Create `connectors.lock.yml` at the repo root.

   ```yaml
   repo: git@github.com:Gemma-Analytics/dlt-connectors.git

   connectors:
     google_ads:
       version: google_ads/v0.1.0
   ```

3. Add `connectors/` to `.gitignore`.

   ```
   # dlt connectors (fetched by sync-connectors.sh, source of truth is connectors.lock.yml)
   connectors/
   ```

4. Run the sync.

   ```bash
   ./sync-connectors.sh
   ```

   This clones each connector at its pinned tag and sets writable permissions on source files. Dependencies are NOT pre-installed — `uv run` installs them on first execution inside the container. Runtime directories (`.venv/`, `.dlt/`, `__pycache__/`) are preserved between syncs to avoid conflicts with files owned by the Docker airflow user.

### Adding a connector

5. Find available connectors and their latest tags.

   ```bash
   git ls-remote --tags git@github.com:Gemma-Analytics/dlt-connectors.git | grep -v '{}' | sort -t/ -k3
   ```

6. Add the connector to `connectors.lock.yml`.

   ```yaml
   connectors:
     google_ads:
       version: google_ads/v0.1.0
     airtable:
       version: airtable/v0.1.0
   ```

7. Sync only the new connector.

   ```bash
   ./sync-connectors.sh airtable
   ```

### Updating a connector

8. Check available tags for the connector.

   ```bash
   git ls-remote --tags git@github.com:Gemma-Analytics/dlt-connectors.git | grep google_ads
   ```

9. Update the version in `connectors.lock.yml`.

   ```yaml
   connectors:
     google_ads:
       version: google_ads/v0.2.0   # bumped from v0.1.0
   ```

10. Re-sync the updated connector.

    ```bash
    ./sync-connectors.sh google_ads
    ```

### Releasing a new connector version (in the monorepo)

11. After merging changes to a connector in the dlt-connectors monorepo, a new tag is created.

    **If auto-tagging is configured** (see the `auto-tag.yml` GitHub Action in the dlt-connectors monorepo): tags are created automatically when changes are merged to `main`. The version bump (patch, minor, major) is determined by the commit message — include `[minor]` or `[major]` in the commit message for non-patch bumps.

    **If tagging manually:**

    ```bash
    # In the dlt-connectors monorepo
    git tag google_ads/v0.2.0
    git push origin google_ads/v0.2.0
    ```

    Check available tags:

    ```bash
    git ls-remote --tags git@github.com:Gemma-Analytics/dlt-connectors.git | grep google_ads
    ```

### CI/CD integration — production deployment

In production, connectors are **baked into the Docker image** at build time. The CI/CD pipeline syncs connectors, then builds the image. The production server never needs git access to the connectors repo.

12. Set up CI/CD to sync connectors before building the Docker image. See `${CLAUDE_SKILL_DIR}/references/cicd-deployment-reference.md` for complete GitHub Actions and GitLab CI workflow templates, deploy key setup, and Dockerfile production stage configuration.

### Using GitLab or other git hosts

13. If a client prefers GitLab (or any other git host), fork or mirror the connectors monorepo there. Update the `repo` URL in `connectors.lock.yml` to the fork URL. See `${CLAUDE_SKILL_DIR}/references/contribute-and-fork-reference.md` for the full GitLab fork workflow.

## Validation

- [ ] `./sync-connectors.sh` runs without errors
- [ ] `connectors/<name>/` directories contain the expected files
- [ ] Connector directories are writable: `ls -la connectors/<name>/` shows `rwx` for all users
- [ ] `uv run python -c "import dlt"` works in each connector directory
- [ ] Versions in `connectors.lock.yml` match the intended tags
- [ ] CI/CD pipeline syncs connectors and builds the image successfully
- [ ] Production image contains connectors at `/opt/airflow/connectors/`

## Examples

**Lockfile with multiple connectors:**

```yaml
repo: git@github.com:Gemma-Analytics/dlt-connectors.git

connectors:
  google_ads:
    version: google_ads/v1.2.0
  airtable:
    version: airtable/v1.0.0
  pipedrive:
    version: pipedrive/v0.3.0
```

**Per-connector version flexibility:**

Each connector is pinned independently. You can update `google_ads` to `v1.3.0` without touching `airtable` at `v1.0.0`. This is a conscious opt-in — no silent updates.

**Contributing back to the monorepo:**

Use `contribute-connector.sh` to submit fixes back to the monorepo as a PR. See `${CLAUDE_SKILL_DIR}/references/contribute-and-fork-reference.md` for the full workflow, including GitLab fork setup.

**Production deployment flow (end-to-end):**

```
Developer pushes to main
  → CI/CD pipeline starts
  → sync-connectors.sh fetches pinned versions (uses git over SSH)
  → docker build bakes connectors/ into the image
  → Image pushed to container registry
  → Production server pulls and runs the image
  → No git access needed on the production server
```