DBT Connector

DBT (data build tool) is a development environment that enables data analysts and data engineers to transform data by simply writing select statements. DBT compiles your code into raw SQL and then runs that code on the specified database and supports collaborative coding patterns and best practices such as version control, documentation, and modularity.

In the OvalEdge application, the DBT connector allows you to crawl the datasets like Projects, JOBS, and RUNS and helps in building the lineage.

Note: The DBT connector is built on the Cloud IDE (integrated development environment) and enables both metadata ingestion and auto lineage

Connector Capabilities

The connectivity to the DBT connector is performed via the API and the versions used by the API are given below:

Technical Specifications

Crawling

Lineage Building

Connection Details

Pre-requisites

1. To use the DBT Connector, the details specified in the following section should be available.
2. By default the service account provided for the connector will be used for any query operations. If the service account has write privileges, then Insert/Update/Delete queries can be executed.

The minimum privileges required are

To connect to the DBT using the OvalEdge application, complete the following steps.

1. Login to the OvalEdge application
2. In the left menu, click on the Administration module name, and the sub-modules associated with the Administration are displayed.
3. Click on the Connectors sub-module name, and the Connectors Information page is displayed.
4. In the Connectors Information page, click on + New Connection. The Manage Connection with Search Connector pop-up window is displayed.
5. In the Manage Connection pop-up window, select the connection type as DBT. The Manage Connection with  DBT specific details pop-up window is displayed.
   
6. The following are the field attributes required for the connection of DBT
   

   Property	Mandatory/ Optional	Details
   Connection Type	Mandatory	By default the connection type is displayed as the DBT, if needed the connection type can be changed by selecting desired connection from the Connection Type dropdown, and based on the selection of the connection type, the fields associated with the selected connection type are displayed.
   License Type	Mandatory	By default the license type is displayed as the Auto Lineage, if needed the license type can be changed by selecting the desired license from the License Type dropdown
   Connection Name	Mandatory	Enter the name of the connection, the connection name specified in the Connection Name textbox will be a reference to the DBT database connection in the OvalEdge application. Example: DBTConnection
   Enter Account Id	Mandatory	Enter the account id associated with the DBT.
   Enter service account token	Mandatory	Enter the account token details associated with the DBT.
   Plugin Server	Optional	Enter the DBT Server details from which the data need to be crawled.
   Plugin Port	Optional	Enter the port number associated with the DBT server.
   Default Governance Roles	Mandatory	Select the required governance roles for the Steward, Custodian, and Owner
   No. of Archive Objects	Mandatory	Enter a count of numbers in the No. of archive objects field. The No. of archive objects count is the number of last modifications made in the metadata data of a Remote/source (for report connection only).  For this, you set the toggle button on the connection page at the time of a crawling connection.   For example, if you update the count as 4 in the ‘No. of archive object’ field, and then the connection is crawled. It will provide the last 4 changes that occurred in the remote/source of the connector. You can observe these changes in the ‘version’ column of the ‘Metadata Changes’ module.
   Select Bridge	Optional	The Bridge ID will be shown in the Bridge dropdown menu when bridges are configured and added, or it will be displayed as "NO BRIDGE".

   Once after entering the connection details in the required fields, click on the button the entered connection details are validated the Save and Save & Configure buttons are enabled.
7. Click on the button to establish the connection or you can also directly click on the button to establish the connection and configure the connection settings. Here when you click on the Save & Configure button, the Connection Settings pop-up window is displayed. Where you can configure the connection settings for the selected Connector. The Save & Configure button is displayed only for the Connectors for which the settings configuration is required.

Error Validations Details

The following are expected errors that can be encountered while establishing the connection.

Connector Settings 

Once establishing the connection successfully the additional configurations for crawling need to be specified. To configure the connector settings for the DBT Connector, select the DBT Connection Name from the Crawler Information page and click on the 9 dots buttons and select the Settings options. The Connection Settings pop-up window is displayed. 

In the Lineage settings, you can select data from multiple servers simultaneously for building the lineage. Here, if more than one server is selected, the lineage will be built by using the first server type, and if the lineage is failed, then lineage will be built by using the second server type, and so on. 

You can configure multiple servers simultaneously in the  Selecting Source Server Type for lineage and also set the connection priority lists to pick the tables in the Connections Priority to pick the source table for lineage building.

FQA’s

1. How to validate the lineage?
   As the DBT supports the internal lineage, the lineage built in the OvalEdge application can be validated just by cross-checking the lineage built in the DBT environment.
   
   
2. Why does DBT not have column lineage?
   As the DBT Rest API does not support the column-level lineage, the OvalEdge also will not have sources to parse and show the column lineages for DBT tables.

Copyright © 2019, OvalEdge LLC, Peachtree Corners GA USA