--- name: setup-1password-dlt-connectors description: Replace Airflow Variables with 1Password CLI (`op run`) for dlt connector secret injection. Use when setting up 1Password service accounts, creating .env.tpl files with op:// references, configuring OP_SERVICE_ACCOUNT_TOKEN, or migrating from Airflow Variable-based secrets to 1Password. disable-model-invocation: true --- # Set Up 1Password for dlt Connectors in Airflow Configure dlt connectors to resolve secrets from 1Password at runtime using `op run --env-file`. This replaces the previous approach of storing secrets in Airflow Variables and writing `secrets.toml` to disk. ## Context Each dlt connector has a `.env.tpl` file containing `op://` secret references. At runtime, `op run --env-file=.env.tpl` resolves these references against 1Password and injects the values as environment variables into the connector subprocess. Secrets never pass through Python memory or touch disk as plaintext. This approach requires: - The 1Password CLI installed in the Airflow Docker image - An `OP_SERVICE_ACCOUNT_TOKEN` passed to the container at runtime - `.env.tpl` files in each connector using bare `op://` references ## Prerequisites - Airflow already set up for uv-based dlt connectors (see `setup-dlt-uv-connector-airflow` skill) - A 1Password service account with read access to the relevant vaults - The service account token stored in 1Password or another secure location ## Steps ### Install 1Password CLI in the Airflow image 1. Add the 1Password CLI to the Dockerfile. This must run as root. ```dockerfile FROM gemmaanalytics/ewah: AS dev_build # For dlt connectors (run via uv with isolated Python) RUN pip install uv # The following commands require root USER root # 1Password CLI for resolving secrets at runtime RUN curl -sS https://downloads.1password.com/linux/keys/1password.asc | \ gpg --dearmor --output /usr/share/keyrings/1password-archive-keyring.gpg && \ echo "deb [arch=amd64 signed-by=/usr/share/keyrings/1password-archive-keyring.gpg] https://downloads.1password.com/linux/debian/amd64 stable main" | \ tee /etc/apt/sources.list.d/1password-cli.list && \ apt-get update && apt-get install -y 1password-cli && \ rm -rf /var/lib/apt/lists/* # Delete default dags from image RUN rm -rf /opt/airflow/dags USER airflow ``` ### Pass the service account token at runtime 2. Add `OP_SERVICE_ACCOUNT_TOKEN` to both the webserver and scheduler services in `docker-compose.yml`. The token is read from the host environment, NOT baked into the image. ```yaml services: webserver: env_file: - .dev_env environment: - OP_SERVICE_ACCOUNT_TOKEN=${OP_SERVICE_ACCOUNT_TOKEN} scheduler: env_file: - .dev_env environment: - OP_SERVICE_ACCOUNT_TOKEN=${OP_SERVICE_ACCOUNT_TOKEN} ``` 3. Start Airflow with the token: ```bash OP_SERVICE_ACCOUNT_TOKEN=$(op item get "" \ --vault "" --fields credential --reveal) \ docker-compose up -d ``` ### Update the shared utility 4. Replace `dags/utils/common.py` with the `op run` version. This removes the Airflow Variable helpers and uses `op run --env-file` instead. ```python """ Shared utilities for dlt connector DAGs. Provides helpers for running connectors via uv subprocess with secrets resolved from 1Password at runtime. """ import os import subprocess CONNECTORS_DIR = "/opt/airflow/connectors" def run_dlt_connector(connector_name, source_name, extra_env=None): """ Run a dlt connector via op run + uv in a subprocess. Uses `op run --env-file` to resolve secrets from 1Password and inject them directly as env vars into the child process. Secrets never pass through Python memory or touch disk as plaintext. Args: connector_name: Directory name under connectors/ source_name: DLT_SOURCE_NAME value extra_env: Optional dict of additional environment variables """ connector_dir = os.path.join(CONNECTORS_DIR, connector_name) env_tpl = os.path.join(connector_dir, ".env.tpl") pipeline_file = f"{connector_name}_pipeline.py" if not os.path.isdir(connector_dir): raise FileNotFoundError(f"Connector directory not found: {connector_dir}") if not os.path.isfile(env_tpl): raise FileNotFoundError(f"Environment template not found: {env_tpl}") # Only op run needs the service account token; # it does NOT forward it to the child process. env = { "PATH": os.environ.get("PATH", "/usr/local/bin:/usr/bin:/bin"), "HOME": os.environ.get("HOME", "/home/airflow"), "OP_SERVICE_ACCOUNT_TOKEN": os.environ["OP_SERVICE_ACCOUNT_TOKEN"], } if extra_env: env.update(extra_env) print(f"Running connector: {connector_name} (source: {source_name})") print(f"Working directory: {connector_dir}") result = subprocess.run( [ "op", "run", f"--env-file={env_tpl}", "--no-masking", "--", "uv", "run", "python", pipeline_file, ], cwd=connector_dir, env=env, capture_output=True, text=True, ) # Print stdout/stderr to Airflow logs if result.stdout: print(result.stdout) if result.stderr: print(result.stderr) if result.returncode != 0: raise RuntimeError( f"Connector {connector_name} failed with exit code {result.returncode}" ) ``` ### Create .env.tpl files for each connector 5. Each connector needs a `.env.tpl` with `op://` references. Use **bare** `op://` syntax (NOT the `{{ op://... }}` format used by `op inject`). **Example for a Snowflake destination with key-pair auth:** ```env # dlt pipeline DLT_DESTINATION=snowflake DLT_SOURCE_NAME= # dlt basic config RUNTIME__DLTHUB_TELEMETRY=false LOAD__DELETE_COMPLETED_JOBS=true # destination DESTINATION__SNOWFLAKE__CREDENTIALS__USERNAME=op:/// - Snowflake DEV - dlt/username DESTINATION__SNOWFLAKE__CREDENTIALS__DATABASE=op:/// - Snowflake DEV - dlt/database DESTINATION__SNOWFLAKE__CREDENTIALS__WAREHOUSE=op:/// - Snowflake DEV - dlt/warehouse DESTINATION__SNOWFLAKE__CREDENTIALS__ROLE=op:/// - Snowflake DEV - dlt/role DESTINATION__SNOWFLAKE__CREDENTIALS__PRIVATE_KEY=op:/// - Snowflake DEV - dlt/private_key DESTINATION__SNOWFLAKE__CREDENTIALS__PRIVATE_KEY_PASSPHRASE=op:/// - Snowflake DEV - dlt/private_key_passphrase DESTINATION__SNOWFLAKE__CREDENTIALS__HOST=op:/// - Snowflake DEV - dlt/account_identifier # sources SOURCES____=op:////field ``` ### Remove Airflow Variables (cleanup) 6. The following Airflow Variables are no longer needed and can be removed: - `dlt_destination` - `dlt_secrets_toml` ```bash docker exec airflow variables delete dlt_destination docker exec airflow variables delete dlt_secrets_toml ``` ### Production deployment 7. Add `OP_SERVICE_ACCOUNT_TOKEN` as a GitHub Actions secret in the Airflow repo (Settings > Secrets and variables > Actions). The CI/CD workflow passes it to the Lightsail deployment as a runtime container environment variable. The deployment script (`create_lightsail_deployment.py`) must include the token in the `environment` dict for both the webserver and scheduler containers: ```python op_token = os.environ["OP_SERVICE_ACCOUNT_TOKEN"] # ... in each container definition: "environment": { "OP_SERVICE_ACCOUNT_TOKEN": op_token, }, ``` The CI/CD workflow must expose the secret to the deployment step: ```yaml env: OP_SERVICE_ACCOUNT_TOKEN: ${{ secrets.OP_SERVICE_ACCOUNT_TOKEN }} ``` Do NOT bake the token into the Docker image as a build arg — it must only be injected at container runtime. ## .env.tpl format rules | Syntax | Correct? | Notes | |--------|----------|-------| | `KEY=op://Vault/Item/field` | Yes | `op run` format | | `KEY={{ op://Vault/Item/field }}` | No | `op inject` format, not compatible with `op run --env-file` | | `KEY=plain_value` | Yes | Non-secret values | | `KEY='{"json": "value"}'` | Yes | Quoted values | ## Validation - [ ] `op` CLI is available in the container: `docker exec op --version` - [ ] Token is set: `docker exec printenv OP_SERVICE_ACCOUNT_TOKEN | head -c 10` shows `ops_...` - [ ] Secrets resolve inside the container: `docker exec op run --env-file=/opt/airflow/connectors//.env.tpl -- printenv ` - [ ] DAG runs successfully end-to-end - [ ] No secrets appear in Airflow Variables - [ ] No `secrets.toml` files exist on disk ## Troubleshooting | Issue | Solution | |-------|----------| | `op: command not found` | 1Password CLI not installed in the image. Add the install step to the Dockerfile. | | `OP_SERVICE_ACCOUNT_TOKEN not set` | Pass the token when starting docker-compose (see step 3). | | `could not resolve secret reference` | Check that the service account has read access to the vault and that the item/field names match exactly. | | `invalid character in secret reference` | Special characters like `:` or `\|` in item titles may cause issues. Rename the item to avoid them. | | `private key decode error` | Ensure the private key is stored as full PEM content (with BEGIN/END markers) or as base64-encoded. Both formats work. | | Secrets not injected (env vars empty) | Check the `.env.tpl` uses bare `op://` syntax, NOT `{{ op://... }}`. |