---
name: containerize-dlt-connector
description: Containerize a dlt connector with Docker — build image, test container, configure secrets encoding. Use when the user wants to Dockerize a connector, create a Dockerfile, build a container, test a Docker build, or prepare a connector for production deployment.
disable-model-invocation: true
---

# Containerize a dlt Connector

Prepare an existing dlt connector for Docker deployment by creating a Dockerfile, building the image, and testing the container.

## Context

Connectors are containerized using a shared base image (`dlt-connectors/dlt-base:latest`) that handles secrets decoding via the `DLT_SECRETS_TOML_BASE64` environment variable. When running in Docker, the database host must be changed from `localhost` to `host.docker.internal`.

## Prerequisites

- Connector works locally with `uv run --env-file=.env python <connector_name>_pipeline.py`
- Dependencies are defined in `pyproject.toml` + `uv.lock`

## Steps

1. Create `Dockerfile` in the connector directory.

   ```dockerfile
   FROM dlt-connectors/dlt-base:latest

   COPY pyproject.toml .
   COPY uv.lock .

   RUN uv sync --frozen

   COPY <connector_name>_pipeline.py .
   # COPY additional files as needed (settings.py, source modules, etc.)

   CMD ["uv", "run", "python", "<connector_name>_pipeline.py"]
   ```

   If the connector has a custom source module directory, add `COPY <connector_name>/ ./<connector_name>/` before the CMD.

2. Identify all files that need to be copied into the container: the pipeline entry point, any source modules, settings files, and transformation files.

3. Update the database host in `secrets.toml` for Docker networking.

   ```bash
   sed -i 's/^host = "localhost"/host = "host.docker.internal"/' \
     connectors/<connector_name>/.dlt/secrets.toml
   ```

4. Build the Docker image from the repository root.

   ```bash
   just build <connector_name>
   ```

   If this fails with "base image not found", run `just build-base-images` first.

5. Run the container with the required environment variables.

   ```bash
   docker run --rm \
     --env DLT_DESTINATION=postgres \
     --env DLT_SOURCE_NAME=<connector_name> \
     --env DLT_SECRETS_TOML_BASE64=$(base64 -w 0 connectors/<connector_name>/.dlt/secrets.toml) \
     dlt-connector-<connector_name>:latest
   ```

   Check the connector's `.env` file for additional required variables and pass them with `--env` flags.

6. Verify the output shows both lines of the success message:

   ```
   Pipeline <name> load step completed in X.XX seconds
   1 load package(s) were loaded to destination postgres
   ```

   To verify the base64-encoded secrets are correct: `echo "$DLT_SECRETS_TOML_BASE64" | base64 -d | head -5`

7. Switch the database host back to localhost for local development.

   ```bash
   sed -i 's/^host = "host.docker.internal"/host = "localhost"/' \
     connectors/<connector_name>/.dlt/secrets.toml
   ```

## Validation

- [ ] Docker build succeeds
- [ ] Container runs and loads data to destination
- [ ] Success message appears in container output
- [ ] Host switched back to `localhost` after testing

## Examples

**Troubleshooting:**

| Issue | Solution |
|-------|----------|
| Module not found | Add missing `COPY` statement to Dockerfile |
| Secrets not found | Verify base64 encoding: `echo "$DLT_SECRETS_TOML_BASE64" \| base64 -d \| head -5` |
| Connection refused | Use `host.docker.internal` instead of `localhost` |
| Base image not found | Run `just build-base-images` first |
| Missing env vars | Check `.env` for vars like `DLT_<SOURCE>_ACCOUNT_IDS` |