dbt module to build DWH on top of myBI Connect data

mybi-dbt-core is dbt module to quick start transformations on top of data synced by myBI Connect along with multiple useful macros.

🚀 Get experience with myBI Market + dbt demo project

Installation instructions

New to dbt packages? Read more about them here.

• Include this package in your packages.yml file
• Run dbt deps to install the package

Main features

Supported Adapters

• Clickhouse
• PostgreSQL

Sources

Reference any data source from sources.yml as select * from source('general', 'dates'):

Models + Tests

Access data tables as simple as select * from ref('stg_yd_ads_facts'):

• filters on account_id applied
• joined with date dimension
• comprehensive data testing (unique, not_null, relationships)

Supported data sources:

• general – General
• metrika – Yandex.Metrika
• direct – Yandex.Direct
• ads – Google Ads
• mytarget – myTarget
• bitrix24 – Bitrix24
• amocrm – AmoCRM
• ga – Google Analytics
• currency – Currency exchange rates
• vkontakte – Vkontakte

Macros

• All the macros and tests from dbt_utils package are available
• source_filter_rows macro to limit rows for specific accounts, dev/test environments
• (WIP) clean_up

Demo project (tutorial)

Get experience with worked example of this module usage:

• 🚀 myBI Market + dbt demo project

Contributing

Development is done via Docker containers and .csv mock files. Refer to integration_tests folder.

Any PR will be tested with Continuous Integration workflow

Development workflow looks like this:

# launch containers: dbt, clickhouse, postgres
docker-compose build --no-cache
docker-compose up -d

# enter container with dbt installed
docker-compose exec dbt bash
cd integration_tests/

# test connections
dbt --version
dbt debug --target clickhouse
dbt debug --target postgres

# Setup target variables
export DBT_SOURCE_DATABASE=default DBT_SOURCE_SCHEMA=default # for Clickhouse
export DBT_SOURCE_DATABASE=postgres DBT_SOURCE_SCHEMA=public # for Postgres

# introduce any code changes

# install dependencies (modules)
dbt clean
dbt deps

# build and test on dummy data
dbt seed --full-refresh
dbt build --full-refresh

# exit container and clean up
docker-compose down