diqu
Automate and streamline the alerting/ notification process for dbt test results using this versatile CLI companion tool. Receive detailed alerts & test metadata seamlessly on various platforms, promoting improved collaboration on dbt project issues .
Who is this for
This tool is designed for individuals or teams seeking to automate the management of their dbt project issues (test warnings, errors... etc) outside the dbt environment.
Features
- Automated alerts and notifications based on recorded dbt test results.
- Built-in support for dq-tools and custom query input.
- Auto-labels
deprecatedtests for quick & easy identification. - Creates and updates Jira tickets with the latest tests' metadata.
- Sends succinct and informative messages to a dedicated Slack channel.
dbt alert rant
"Run errors are red,
Test warnings are yellow,
Where's my bug alert,
And joint bugfix workflow?"
-- bydiquOG Contributors
Supported Modules
- Sources (DWH connections)
- Snowflake
- CSV file
- Package (parsing dbt test results)
- Alert Modules (alert/ notification)
- Jira
- Slack
Installation
NOTE: The required Data Warehouse (DWH) module should already be installed if you are using diqu in a working dbt project. If not, please perform additional steps to install these DWH modules.
For example, if you're using Snowflake:
pip install "snowflake-connector-python[secure-local-storage]"
Usage
preflight
export ISSUE_DEPRECATED_WINDOW_IN_DAYS=your_issue_deprecation_time_ in_day, default to "3"
export ISSUE_UPDATE_WINDOW_IN_DAYS=your_issue_historical_data_updat e_window_in_days, default to "14"
# build dq-tools log table
dbt run -s dq_tools
04:33:17 diqu: INFO - Run with diqu==1.0.0
04:33:19 diqu: INFO - Using dbt project at: /path/to/dbt/project
04:33:19 diqu: INFO - Using dbt profiles.yml at: ~/.dbt
04:33:19 diqu: INFO - Using snowflake connection
04:33:19 diqu: INFO - Looking for the query in: ./dq_tools__get_test_results.sql
04:33:23 diqu: INFO - Alerting to: SLACK
04:33:23 diqu: INFO - Done > Slack
04:33:23 diqu: INFO - Alerting to: JIRA
04:33:23 diqu: INFO - Done > JIRA
Alert Modules Configurations
Slack
-
Use the environment variables to configure the Slack Channel:
export SLACK_TOKEN=your_token
export SLACK_CHANNEL=your_channel_name
diqu alert --to slack
Jira Board
-
Use the environment variables to configure the JIRA Board:
export JIRA_SERVER=your_jira_server e.g. https://your_value.atlassian.net/
export JIRA_AUTH_USER=your_service_account e.g. dqt_user@your_value.com
export JIRA_AUTH_PASSWORD=your_service_token e.g. ATATTxxxxx
export JIRA_PROJECT_ID=your_project_id e.g. 106413
export JIRA_ISSUE_TYPE=your_issue_type, default to "Bug"
export JIRA_OPEN_ISSUES_FILTER_BY_SUMMARY=your_issue_filter_on_titl e, default to "dq-tools"
diqu alert --to jira
For more details, please visit the documentation site.
Quick Demo
How to Contribute
This Auto Alert (diqu) tool is an open-source software. Whether you are a seasoned open-source contributor or a first-time committer, we welcome and encourage you to contribute code, documentation, ideas, or problem statements to this project.
See CONTRIBUTING guideline for more details or check out CONTRIBUTING.md
And then, kudos to our beloved Contributors:
About Infinite Lambda
Infinite Lambda is a cloud and data consultancy. We build strategies, help organizations implement them, and pass on the expertise to look after the infrastructure.
We are an Elite Snowflake Partner, a Platinum dbt Partner, and two-times Fivetran Innovation Partner of the Year for EMEA.
Naturally, we love exploring innovative solutions and sharing knowledge, so go ahead and:
Take a look around our Git
Browse our tech blog
We are also chatty, so:
Follow us on LinkedIn
Or just get in touch