---
name: update-airflow3-connectors
description: Update or add dlt connectors in an Airflow 3 template repo's connectors/ folder by diffing local hard copies against the dlt-connectors monorepo main branch and interactively merging changes. Use when updating existing connectors like fx or pokemon to the latest upstream version, reviewing local customizations against upstream, or adding new connectors from the monorepo to an Airflow 3 repository.
disable-model-invocation: true
argument-hint: "[connector-name]"
---

# Update Airflow 3 Connectors

Update the hard-copied dlt connectors in an Airflow 3 template repository (`connectors/` subfolder) from the [dlt-connectors monorepo](https://github.com/Gemma-Analytics/dlt-connectors), preserving deliberate local customizations, and optionally add new connectors.

**Fast path:** when invoked with a connector name (`$ARGUMENTS`), skip the full update review — run the check script (still needed for the clone), then go directly to that connector: add it (step 5) if not local, or show only its diff (steps 2–4) if it is. Do not narrate the other connectors' findings.

The check always runs before any question — it is read-only, cheap, and both flows (update + add) need its clone and output. "Update or add?" is only answerable with the diff in hand; often the honest answer is "nothing to update".

## Context

In the Airflow 3 template pattern ([airflow3-best-practice](https://github.com/Gemma-Analytics/airflow3-best-practice)), connectors are **hard copies** — self-contained uv projects under `connectors/<name>/`, bind-mounted into Airflow. There is no lockfile or sync script; local copies may carry deliberate custom logic. Updating therefore requires a **diff-aware, interactive merge**, not a blind overwrite.

**Check the repo shape before running** — three shapes exist and only one belongs to this skill:

| Repo shape | Markers | Skill |
|---|---|---|
| Airflow 3 template | `connectors/` folder, `dags/utils/dlt.py` | **this skill** |
| Airflow 2, lockfile-distributed | `connectors.lock.yml` + `sync-connectors.sh` (one-way overwrite) | `gemma-dlt/sync-dlt-connectors` |
| Airflow 2 + EWAH/airflowprovider, no `connectors/` folder | in-process operators (e.g. `dltOperator`), old pinned dlt | `setup-dlt-uv-connector-airflow` (modern hard-copy variant) — sets the repo up first; afterwards this skill handles updates |

## Prerequisites

- An Airflow 3 template repository with a `connectors/` subfolder
- Git access to `git@github.com:Gemma-Analytics/dlt-connectors.git`
- Working tree clean enough to review a merge (commit or stash unrelated changes first)

**Execution notes (token/roundtrip economy):**

- Use absolute paths in every command — shell working directories may not persist between tool calls
- Never read the `.dlt` example TOML **contents** — their filenames are deny-listed in Gemma environments (predictable block + recovery cost), and they're not needed: credential requirements are visible as `dlt.secrets.value` parameters in the pipeline source and in the README. Compare example TOMLs only by checksum (the check script does this)
- Batch related checks into compound commands where possible instead of one command per check

## Steps

### 1. Run the check script

The mechanical part — fresh shallow clone of upstream main, per-connector diff with artifact exclusions, CRLF neutralization, example-TOML checksum comparison, new-connector listing — is one deterministic script (run from the Airflow repo root):

```bash
bash "${CLAUDE_SKILL_DIR}/scripts/update_check.sh"
# options: --connectors-dir <path>   (default: connectors)
#          --repo <git-url>          (default: dlt-connectors monorepo)
```

It makes no changes and emits a compact summary: `UPSTREAM_CLONE=<path>` (the clone is kept for the follow-up merge/copy steps), then per local connector `IDENTICAL` / `DIFFERS` (with `changed:` / `local-only:` / `upstream-only:` / `example-toml …` findings) / `NOT_UPSTREAM`, then the list of upstream connectors not present locally. Only local connectors are in scope for the update phase.

**Token economy:** connectors reported `IDENTICAL` need no further attention or narration. Show full content diffs only for files the user must decide on:

```bash
diff -u --strip-trailing-cr "connectors/<name>/<file>" "$UPSTREAM_CLONE/connectors/<name>/<file>"
```

(Manual fallback if the script is unavailable: `diff -rq --strip-trailing-cr` with `--exclude` flags for `.venv`, `__pycache__`, `*.duckdb`, `*.wal`, `.dlt`, `.cursor`, `.e*`. Never use `git diff --no-index` with `:(exclude)` pathspecs — git rejects the extra pathspecs with a usage error — and never write the real credential-file names into commands; compare only `*_example.toml`, by normalized checksum.)

### 2. Classify the findings

Classify every differing file into one of three groups and present a per-connector summary to the user:

| Group | Meaning |
|---|---|
| Upstream-only change | Upstream evolved; local copy is behind |
| Local customization | Local copy has deliberate changes not upstream |
| Both changed | Needs a real merge decision |

Show actual diffs (not just file lists) for anything beyond trivial size.

### 3. Ask the user what to merge

**If nothing changed** (all connectors `IDENTICAL`, or only differences already decided in this session/repo history): do NOT walk the merge flow — state it in one line and ask a single compact question: add a connector, open the pending upstream-drift PR (if any), or done.

For each difference group, ask the user explicitly — do not assume:

- **Take upstream** — overwrite the local file(s)
- **Keep local** — skip the upstream change
- **Merge selectively** — apply specific hunks/files; walk through them one by one

Rules during the merge:

- `uv.lock` travels as a unit with `pyproject.toml`: if `pyproject.toml` is taken from upstream, take `uv.lock` with it; never hunk-merge a lockfile
- Never copy real `secrets.toml` / `config.toml` in either direction — and **never delete them locally either**: when taking upstream (or re-copying a connector), overwrite tracked files individually rather than `rm -rf` + copy. A wholesale replace silently destroys the operator's local runtime files in `.dlt/` (credentials, settings) and breaks working setups
- If a local customization is worth keeping long-term, remind the user it should be contributed upstream (see the `contribute-dlt-connector` skill in the `gemma-dlt` plugin) — copies should not drift silently

### 4. Post-merge checks per updated connector

- The standard env-var pattern is intact in `<name>_pipeline.py`:
  ```python
  DLT_DESTINATION = os.getenv("DLT_DESTINATION", "duckdb")
  DLT_SOURCE_NAME = os.getenv("DLT_SOURCE_NAME", "<connector_name>")
  DLT_DATASET_NAME = os.getenv("DLT_DATASET_NAME", DLT_SOURCE_NAME)
  DLT_PIPELINES_DIR = os.getenv("DLT_PIPELINES_DIR")
  ```
  with `dataset_name=DLT_DATASET_NAME, pipelines_dir=DLT_PIPELINES_DIR` passed to `dlt.pipeline()`. Re-add if upstream lacks it.
- The per-connector `.gitignore` still whitelists `.dlt/*_example.toml` and ignores `.env`, `__pycache__/`, `*.duckdb`
- `uv lock --check` passes in the connector directory

### 5. Offer new connectors

The script's `== new upstream connectors ==` section already lists upstream connectors not present locally — present it and let the user select any to add. For each selected connector:

1. `ls -a` the source dir — **that is all the pre-copy inspection needed** (it exists only to apply the exclusions; upstream dirs have historically contained committed credential TOMLs and duckdb files). Do not read connector code before copying.
2. Copy from `$UPSTREAM_CLONE`, then strip with hook-safe globs — never write the real credential-file names into the command (the secrets-policy hook blocks them):

   ```bash
   cp -r "$UPSTREAM_CLONE/connectors/<name>" connectors/
   rm -rf connectors/<name>/.venv connectors/<name>/.cursor
   find connectors/<name> -type d -name __pycache__ -exec rm -rf {} +
   find connectors/<name> -type f \( -name '*.duckdb' -o -name '*.wal' \) -delete
   find connectors/<name>/.dlt -type f ! -name '*_example.toml' -delete
   ```
3. Apply the step 4 checks (env vars, `.gitignore`, `uv lock --check`) — this is where reading the pipeline file belongs, after the copy.

An added connector almost always wants a DAG next, but DAG creation is a separate user-triggered skill (it cannot be auto-chained). **End every add-flow by telling the user the exact next command**, one per added connector:

```
/gemma-airflow:create-airflow3-connector-dag <connector_name>
```

### 6. Clean up

```bash
rm -rf "$UPSTREAM_CLONE"
```

Summarize for the user: per connector, what was taken from upstream, what local logic was kept, and what was newly added.

### 7. Commit and open a PR

Never commit to `main` — work on a feature branch (`feat/update-connectors-<date>` or the repo's convention). Then, **after confirming with the user**:

```bash
git checkout -b <feature_branch>   # skip if already on one
git add connectors/
git commit -m "feat: update connectors from dlt-connectors main (<summary>)"
git push -u origin <feature_branch>
gh pr create --fill
```

- One commit per logical change is fine (updates vs newly added connectors may be separate commits)
- Verify `git status` shows no real credential files staged before committing
- If local customizations were kept, offer to open the corresponding **upstream dlt-connectors PR** in the same session (`contribute-dlt-connector` skill) so the drift gets resolved at the source
- For broader end-of-session git wrap-up, see the `wrap` skill in the `gemma-devx` plugin

## Validation

- [ ] `docker compose exec airflow-scheduler airflow dags list-import-errors` shows no errors
- [ ] Each DAG whose connector changed was re-triggered and ran to success (check task logs for row counts)
- [ ] `uv lock --check` passes in every touched connector directory
- [ ] No real `secrets.toml` / `config.toml` / `.venv` / `*.duckdb` entered version control (`git status`)

## Examples

**Typical update session** (script output → decisions):

```text
UPSTREAM_CLONE=/tmp/tmp.Xa12bC
UPSTREAM_COMMIT=c3dfa65
== existing connectors ==
connector fx: IDENTICAL
connector pokemon: DIFFERS
  upstream-only: config.py
connector sql_database: DIFFERS
  changed: sql_database_pipeline.py
== new upstream connectors ==
airtable
hubspot
pipedrive
...

→ fx: skipped silently (identical)
→ pokemon: user keeps local (dead file deliberately dropped)
→ sql_database: local customization kept → offer upstream PR
→ user selects pipedrive → copied + checks applied → create-airflow3-connector-dag
```

**Related skills:** `setup-airflow3-dlt-template` (initial template setup) · `create-airflow3-connector-dag` (DAG for a copied connector) · `gemma-dlt/contribute-dlt-connector` (upstream a local customization) · `gemma-dlt/sync-dlt-connectors` (Airflow 2 lockfile pattern)