# dbt-invoke dbt-invoke-gemma is a CLI for creating, updating, and deleting [dbt](https://docs.getdbt.com/docs/introduction) [property files](https://docs.getdbt.com/reference/declaring-properties) and [md files](https://docs.getdbt.com/docs/build/documentation#using-docs-blocks). Its primary purpose is to assist in generating [dbt documentation](https://docs.getdbt.com/docs/build/documentation). This package is a Gemma adaptation of the [dbt-invoke]() CLI (built with [Invoke](http://www.pyinvoke.org/)). - Supported dbt resource types: - models - snapshots - analyses - Under the hood, this tool works by combining the power of the [dbt ls](https://docs.getdbt.com/reference/commands/list) and [dbt run-operation](https://docs.getdbt.com/reference/commands/run-operation) commands with dbt's built-in `get_columns_in_query` macro. - This methodology allows the tool to work on [ephemeral models](https://docs.getdbt.com/docs/building-a-dbt-project/building-models/materializations#ephemeral) and [analyses](https://docs.getdbt.com/docs/building-a-dbt-project/analyses), which other approaches, such as those based on listing data warehouse tables/views, can miss. ## Installation ```shell pip install dbt-invoke-gemma ``` **NOTE: dbt version 1.8 or above is required** ## Usage - There are 2 main commands - `dbt-invoke --mdfiles`: generate md files with doc blocks. - `dbt-invoke --properties`: generate property files referencing the doc blocks. - You must have previously executed `dbt run`/`dbt seed`/`dbt snapshot` on the resources for which you wish to create/update property files. - If you have made updates to your resources, execute the appropriate command (`dbt run`/`dbt seed`/`dbt snapshot`) before using this tool to create/update property files. - The package also provides suggested prompts to feed to the CursorAI agent to automatically generate documentation. To use the prompts you must have a valid cursor account. ### Generate md files ```shell dbt-invoke mdfiles ``` - The first time you run this command, you should be prompted to add a short macro called `_log_columns_list` to your dbt project. - You may accept the prompt to add it automatically. - Otherwise, copy/paste it into one your dbt project's macro-paths yourself. **NOTE: You need to generate md files BEFORE generating the property files, otherwise the system will throw an error.** - md files will be created, updated, or deleted on a one-to-one basis in a subfolder of the path of the resource files they represent (naming convention: `_{resource folder name}_docs`) - For example, given a resource file in the location `models/marts/core/users.sql`, this tool will create, update, or delete a property file in the location `models/marts/core/_core_docs/users.md`. If the docs subfolder (ex.`models/marts/core/_core_docs`) does not exist, it will be generated by the package. - Any newly generated md files are created with the doc snippets for the columns of the resource (naming convention: `{model name}__{COLUMN NAME}`). - For example, when generating a new property file for a model `users` with column names `user_id` and `created_at`, the following md file will be generated: - ```md {% docs users__USER_ID %} {% enddocs %} {% docs users__CREATED_AT %} {% enddocs %} ``` - When updating an already existing md file, existing columns and descriptions will remain as they are, new columns in the resource will be added (however, columns that no longer exist will not be removed). - The command will also provide a suggestion on how to prompt the CursorAI agent to automatically fill out the content of the doc files. ``` To add the documentation to fields, use the following prompt for cursor AI after passing the following as context: - the md file of the model - the sql file of the model > Check the query for the sql model of this md file, and fill out the empty doc snippets > (only the empty doc snippets, do not edit the ones that already have content). > For each field the info should include: > - description > - column level lineage > - calculation or field logic (if the field is derived) > Check this file well, it is really important that the info is correct. > Also, fill out ALL empty doc snippets, do not skip any fields. > Give me the change immediately, do not wait for me to ask several times to change the file ``` ### Generate property files ```shell dbt-invoke properties ``` - Property files will be created, updated, or deleted on a one-to-one basis in a subfolder of the path of the resource files they represent (naming convention: `_{resource folder name}_schemas`) - For example, given a resource file in the location `models/marts/core/users.sql`, this tool will create, update, or delete a property file in the location `models/marts/core/_core_schemas/users.yml`. - If your dbt project defines properties for multiple resources per `.yml` file, see the [Migrating to One Resource Per Property File](#migrating-to-one-resource-per-property-file) section. - Any newly generated property files are created with the correct resource type, resource name, and columns. For each column, the related doc snippet (naming convention: `{model name}__{COLUMN NAME}`) will be added as description. - For example, when generating a new property file for a model `users` with column names `user_id` and `created_at`, the following yaml will be generated: - ```yaml version: 2 models: - name: users description: '' columns: - name: user_id description: |- {{ doc("users_USER_ID") }} data_tests: [] - name: created_at description: |- {{ doc("users_CREATED_AT") }} data_tests: [] ``` - When updating an already existing property file, new columns in the resource will be added, and columns that no longer exist will be removed. - You may add other properties (e.g. `data tests`). They will remain intact when updating existing property files as long as the column/resource name to which they belong still exists. ### Options `` primarily uses the same arguments as the `dbt ls` command to allow flexibility in selecting the dbt resources for which you wish to create/update property files (run `dbt ls --help` for details). - --resource-type - --models - --select - --selector - --exclude - --project-dir - --profiles-dir - --profile - --target - --vars - --bypass-cache - --state - Notes: - This tool supports only the long flags of `dbt ls` options (for example: `--models`, and not short flags like `-m`). - Multiple values for the same argument can be passed as a comma separated string (Example: `--models modelA,modelB`) - Keep in mind that dbt may not support multiple values for certain options. - Two additional flags are made available. - `--log-level` to alter the verbosity of logs. - It accepts one of Python's standard logging levels (debug, info, warning, error, critical). - `--threads` to set a maximum number of concurrent threads to use in collecting resources' column information from the data warehouse and in creating/updating the corresponding property files. Each thread will run dbt's get_columns_in_query macro against the data warehouse. - Some examples: ```shell # Create/update md files for all supported resource types dbt-invoke mdfiles # Create/update property files for all supported resource types dbt-invoke properties # Create/update md files for all supported resource types, using 4 concurrent threads dbt-invoke mdfiles --threads 4 # Create/update property files for all supported resource types, using 4 concurrent threads dbt-invoke properties --threads 4 # Create/update md files for all models in a models/marts directory dbt-invoke mdfiles --models marts # Create/update property files for all models in a models/marts directory dbt-invoke properties --models marts # Create/update md files for a 'users' model and an 'orders' models dbt-invoke mdfiles --models users,orders # Create/update property files for a 'users' model and an 'orders' models dbt-invoke properties --models users,orders # Create/update md files for a 'users' model and all downstream models dbt-invoke mdfiles --models users+ # Create/update property files for a 'users' model and all downstream models dbt-invoke properties --models users+ # Create/update a md file for a snapshot called 'users_snapshot' dbt-invoke mdfiles --resource-type snapshot --select users_snapshot # Create/update a property file for a snapshot called 'users_snapshot' dbt-invoke properties --resource-type snapshot --select users_snapshot ``` ### Deleting Property Files ```shell dbt-invoke properties.delete ``` - `` uses the same arguments as for creating/updating property files, except for `--threads`. ### Help - To view the list of available commands and their short descriptions, run: ```shell dbt-invoke --list ``` - To view in depth command descriptions and available options/flags, run: ```shell dbt-invoke --help ``` ### Limitations - dbt-invoke will try to preserve formatting and comments when updating existing files. If you want to preserve line-breaks, use `>` or `|` on your multiline strings, as recommended [here](https://stackoverflow.com/a/21699210/15202709). - In order to collect or update the list of columns that should appear in each property file, dbt's `get_columns_in_query` macro is run for each matching resource. As of the time of writing, `get_columns_in_query` uses a SELECT statement [limited to zero rows](https://github.com/fishtown-analytics/dbt/blob/2b48152da66dbd7f07272983bbc261f1b6924f20/core/dbt/include/global_project/macros/adapters/common.sql#L11). While this is not typically a performance issue for table or incremental materializations, execution may be slow for complex analyses, views, or ephemeral materializations. - This may be partially remedied by increasing the value of the `--threads` option in `dbt-invoke properties.update`. - dbt-invoke has not been tested across different types of data warehouses.