---
name: debug-dlt-connector
description: Diagnose and fix dlt connector failures — authentication errors, empty results, broken pagination, incremental loading issues, Docker networking problems. Use when the user mentions an error, failure, 401/403, no data loaded, pagination stops, connector not working, or needs to troubleshoot logs.
---

# Debug a dlt Connector

Common debugging approaches for dlt connectors when they fail, return no data, or have authentication/pagination issues.

## Context

dlt connectors can fail at multiple levels: source configuration, authentication, pagination, incremental state, or Docker networking. This skill covers systematic debugging for each failure mode.

## Steps

1. Enable verbose logging to see detailed pipeline output.

   ```bash
   RUNTIME__LOG_LEVEL=DEBUG uv run --env-file=.env python <connector_name>_pipeline.py
   ```

   Or set in `secrets.toml`:

   ```toml
   [runtime]
   log_level = "DEBUG"
   ```

2. Test the source independently without running the full pipeline.

   ```python
   if __name__ == "__main__":
       source = my_source()
       for resource in source:
           print(f"Resource: {resource.name}")
           count = 0
           for record in resource:
               print(record)
               count += 1
               if count >= 3:
                   break
           print(f"  ... {count}+ records")
   ```

3. For authentication issues, verify the secrets.toml structure matches the expected path.

   ```toml
   [sources.<DLT_SOURCE_NAME>]
   api_token = "..."
   ```

   Check that the token format is correct (some APIs need "Bearer " prefix) and that the token hasn't expired.

4. For empty results, check that `data_selector` matches the API response structure by testing the API manually.

   ```python
   import requests
   resp = requests.get("https://api.example.com/endpoint", headers={...})
   print(resp.json())
   ```

5. For pagination issues, verify the paginator configuration matches API docs and add dlt-level debug logging.

   ```python
   import logging
   logging.getLogger("dlt").setLevel(logging.DEBUG)
   ```

6. For incremental loading not working, inspect the pipeline state.

   ```python
   pipeline = dlt.pipeline(...)
   print(pipeline.state)
   ```

   Verify that `write_disposition` is `"merge"` (not `"replace"`) and that `primary_key` is defined.

7. For issues when running inside Airflow, exec into the scheduler container and test directly.

   ```bash
   docker exec -it <scheduler_container> bash
   cd /opt/airflow/connectors/<connector_name>
   uv run python <connector_name>_pipeline.py
   ```

8. Use DuckDB for quick testing without a database. Set `DLT_DESTINATION=duckdb` in `.env`, then inspect:

   ```python
   import duckdb
   conn = duckdb.connect("<source_name>.duckdb")
   conn.sql("SHOW TABLES").show()
   conn.sql("SELECT * FROM <table> LIMIT 10").show()
   ```

9. For interactive debugging, add ipdb breakpoints:

   ```bash
   uv add ipdb --dev
   # Add to code: import ipdb; ipdb.set_trace()
   uv run --env-file=.env python <connector_name>_pipeline.py
   ```

10. Inspect pipeline state programmatically:

    ```python
    import dlt
    pipeline = dlt.pipeline(pipeline_name="<name>", destination="duckdb")
    print(pipeline.last_trace)
    print(pipeline.state)
    print(pipeline.default_schema.tables.keys())
    ```

11. To start fresh, clean the pipeline state.

    ```bash
    rm -rf ~/.dlt/pipelines/<pipeline_name>
    rm -f <source_name>.duckdb*
    ```

## Validation

- [ ] Root cause of failure identified
- [ ] Fix applied and connector runs successfully
- [ ] Verified data loads correctly after fix

## Examples

**Common issues and solutions:**

| Issue | Symptoms | Solution |
|-------|----------|----------|
| No data loaded | Pipeline runs but 0 rows | 1. Check API response manually with `requests.get()` 2. Verify `data_selector` path matches JSON structure 3. Check pagination config |
| Auth failed | 401 or 403 errors | Verify token format and expiry |
| Connection refused in Docker | Works locally, fails in container | Use `host.docker.internal` |
| Module not found in Docker | ImportError on run | Add missing `COPY` to Dockerfile |
| Pagination stops early | Only first page loaded | Check paginator `cursor_path` or `total_path` |
| Incremental reloads everything | Full data every run | Check `write_disposition`, `primary_key`, cursor field |

**Inspecting destination data (Postgres):**

```bash
# List tables
PGPASSWORD=dev12345_ psql -h localhost -p 5434 -U dlt -d postgres -c "
SELECT table_name FROM information_schema.tables
WHERE table_schema = '<connector_name>' ORDER BY table_name;"

# Check dlt load metadata
PGPASSWORD=dev12345_ psql -h localhost -p 5434 -U dlt -d postgres -c "
SELECT * FROM <connector_name>._dlt_loads ORDER BY inserted_at DESC LIMIT 5;"
```