DBT Connector

DBT (data build tool) is a development environment that enables data analysts and data engineers to transform data by simply writing select statements. DBT compiles your code into raw SQL and then runs that code on the specified database and supports collaborative coding patterns and best practices such as version control, documentation, and modularity.

In the OvalEdge application, the DBT connector allows you to crawl the datasets like Projects, JOBS, and RUNS and helps in building the lineage.

Note: The DBT connector is built on the Cloud IDE (integrated development environment) and enables both metadata ingestion and auto lineage

DBTConnection

Connector Capabilities

The connectivity to the DBT connector is performed via the API and the versions used by the API are given below:

Driver/API

Version

Details

API

V2

https://docs.getdbt.com/docs/dbt-cloud-apis/metadata-api

Technical Specifications

Crawling

Feature

Supported Objects

Remarks

Crawling

DBT Projects

Show all the projects from DBT for the specified account id

DBT RUNS

Runs from DBT

DBT JOBS

Jobs from DBT

Lineage Building

Feature

Supported Objects

Remarks

Lineage

DBT RUN

Supported

DBT JOB

Not Supported

Connection Details

Pre-requisites

  1. To use the DBT Connector, the details specified in the following section should be available.
  2. By default the service account provided for the connector will be used for any query operations. If the service account has write privileges, then Insert/Update/Delete queries can be executed.

The minimum privileges required are

Operation 

Access Permission

Connection Validation

Read Only Access 

Crawling

Read Only Access 

To connect to the DBT using the OvalEdge application, complete the following steps.

  1. Login to the OvalEdge application
  2. In the left menu, click on the Administration module name, and the sub-modules associated with the Administration are displayed.
  3. Click on the Connectors sub-module name, and the Connectors Information page is displayed.
  4. In the Connectors Information page, click on + New Connection. The Manage Connection with Search Connector pop-up window is displayed.
  5. In the Manage Connection pop-up window, select the connection type as DBT. The Manage Connection with  DBT specific details pop-up window is displayed.
    DBT
  6. The following are the field attributes required for the connection of DBT

    Property

    Mandatory/ Optional

    Details

    Connection Type

    Mandatory

    By default the connection type is displayed as the DBT, if needed the connection type can be changed by selecting desired connection from the Connection Type dropdown, and based on the selection of the connection type, the fields associated with the selected connection type are displayed.

    License Type

    Mandatory

    By default the license type is displayed as the Auto Lineage, if needed the license type can be changed by selecting the desired license from the License Type dropdown

    Connection Name

    Mandatory

    Enter the name of the connection, the connection name specified in the Connection Name textbox will be a reference to the DBT database connection in the OvalEdge application.

    Example: DBTConnection

    Enter Account Id

    Mandatory

    Enter the account id associated with the DBT.

    Enter service account token

    Mandatory

    Enter the account token details associated with the DBT.

    Plugin Server

    Optional

    Enter the DBT Server details from which the data need to be crawled.

    Plugin Port

    Optional

    Enter the port number associated with the DBT server.

    Default Governance Roles

    Mandatory

    Select the required governance roles for the Steward, Custodian, and Owner

    No. of Archive Objects 

    Mandatory

    Enter a count of numbers in the No. of archive objects field. The No. of archive objects count is the number of last modifications made in the metadata data of a Remote/source (for report connection only). 

    For this, you set the toggle button on the connection page at the time of a crawling connection.  

    For example, if you update the count as 4 in the ‘No. of archive object’ field, and then the connection is crawled. It will provide the last 4 changes that occurred in the remote/source of the connector. You can observe these changes in the ‘version’ column of the ‘Metadata Changes’ module. 

    Select Bridge

    Optional

    The Bridge ID will be shown in the Bridge dropdown menu when bridges are configured and added, or it will be displayed as "NO BRIDGE". 

    Once after entering the connection details in the required fields, click on the button the entered connection details are validated the Save and Save & Configure buttons are enabled.
  7. Click on the button to establish the connection or you can also directly click on the button to establish the connection and configure the connection settings. Here when you click on the Save & Configure button, the Connection Settings pop-up window is displayed. Where you can configure the connection settings for the selected Connector. The Save & Configure button is displayed only for the Connectors for which the settings configuration is required.

Error Validations Details

The following are expected errors that can be encountered while establishing the connection.

S.No.

Error Messages

Description

1.

Account Id Incorrect

The provided DBT Account details are not valid.

2.

Service Account Token

The provided DBT Service Account Token details are not valid.

Connector Settings 

Once establishing the connection successfully the additional configurations for crawling need to be specified. To configure the connector settings for the DBT Connector, select the DBT Connection Name from the Crawler Information page and click on the 9 dots buttons and select the Settings options. The Connection Settings pop-up window is displayed. 

DBT Settings

In the Lineage settings, you can select data from multiple servers simultaneously for building the lineage. Here, if more than one server is selected, the lineage will be built by using the first server type, and if the lineage is failed, then lineage will be built by using the second server type, and so on. 

You can configure multiple servers simultaneously in the  Selecting Source Server Type for lineage and also set the connection priority lists to pick the tables in the Connections Priority to pick the source table for lineage building.

FQA’s

  1. How to validate the lineage?
    As the DBT supports the internal lineage, the lineage built in the OvalEdge application can be validated just by cross-checking the lineage built in the DBT environment.

  2. Why does DBT not have column lineage?
    As the DBT Rest API does not support the column-level lineage, the OvalEdge also will not have sources to parse and show the column lineages for DBT tables.

Copyright © 2019, OvalEdge LLC, Peachtree Corners GA USA