Skip to content

dbt_artifacts.md

Latest commit

 

History

History
90 lines (62 loc) · 2.89 KB

dbt_artifacts.md

File metadata and controls

90 lines (62 loc) · 2.89 KB
Raw

Custom Artifact Generation (dbt)

We have a handful of custom scripts that build various artifacts required by different components of the various data pipeline components. These should all be handled by the CI/CD pipeline and added to the release's Assets list.

  • The SQL Alchemy model is available directly from the assets list as a whl file and can be added to your python distribution using @git+{url} syntax.
  • The all artifacts generated by the just site recipe can be found inside the release assets' project_artifacts.zip file.

To trigger the production of these assets, a new release should be created with a semantic version based tag that starts with v: v1.0.0, etc.

The dbt Recipe

A dumb wrapper recipe has been created with dependencies to trigger the production of each of those dbt related artifacts.

This recipe has been added as a dependency of the standard recipe, site.

just dbt

Example command to add the model to your uv based python project:

uv add common_access_model@https://github.com/include-dcc/common-access-model/releases/download/0.0.1/common_access_model-0.0.0.post89.dev0+d378cee-py3-none-any.whl

Please note the releases/download/0.0.1 portion of the url. That is to a specific version of the model itself, which is by design. You will need to update your project each time you want to update to a new model.

In a similar fashion, you can install it via pip or add it to your pyproject.toml file using a similar pattern.

SQL Alchemy Model

Many of our tools rely on the SQL Alchemy models produced using LinkML's SQL Alchemy generator. We've added just recipe's to run build these:

just sqla

The resulting file is found inside project/sqlalchemy/{model_name}.py

Data Dictionary csv files

The pipeline utitilies created by the FTD team rely on csv data dictionaries to the target .sql files.

just _gen_ftddd

Resulting csv files can be found inside: project/data-dictionary/*.csv

Each class from the model and any of the SQL specific join tables will be found in their own file, conforming to the FTD DD Format.

Model Enums

This is a CSV file that contains each enumeration and it's home within the datamodel. It is used by the dbt pipeline utitilies. This is a secondary artifact from the _gen_ftddd recipe.

The resulting file is found inside project/enumerations/model_enums.csv

dbt Model

A complete model that can be immediately dropped into your dbt pipeline can be generated that features various dbt tests based on the model's definition including:

  • valid_uri_format
  • foreign key target must exist
  • uri-or-curie
  • multi-column keys
  • not null
  • unique
  • allowable values

The resulting file is found inside project/dbt/src_dev_include_access.yml

The filename is purely a placeholder and should be renamed to whatever makes sense for your needs.