Knowledge Base

Connectors

ETLs

DBT Core

DBT Core (data build tool) is a development environment that enables data analysts and data engineers to transform data by simply writing select statements. In the OvalEdge application, the DBT core connector allows users to crawl the datasets like Projects, JOBS, and RUNS and helps in building the lineage.

DBT Core

Connector CapabilitiesTechnical Specifications

Crawling

Feature	Supported Objects	Remarks
Crawling	DBT Projects	Show all the projects from DBT for the specified account id
Crawling	DBT RUNS	RUNS from DBT

Lineage Building

Feature	Supported Objects	Remarks
Lineage	DBT RUN	Supported
Lineage	DBT JOB	Not Supported

Connection Details

Pre-requisites

To use the DBT Core Connector, the details specified in the following section should be available.

The minimum privileges required are:

Operation	Access Permission
Connection Validation	Read Only Access
Crawling	Read Only Access

Configuring Environment Variables

Configuring environment names enable you to select the appropriate environment from the drop-down list when adding a connector. This allows for consistent crawling of schemas across different environments, such as production (PROD), staging (STG), or temporary environments. It also facilitates schema comparisons and assists in application upgrades by providing a temporary environment that can be later deleted if needed.

Before establishing a connection, it is important to configure the environment names for the specific connector. If your environments have been configured, skip this step.

Steps to Configure the Environment

Log into the OvalEdge application.
Navigate to Administration > System Settings.
Select the Connector tab.
Find the key name “connector.environment”.
Enter the desired environment values (PROD, STG) in the Value column.
Click ✔ to Save.

Establish a Connection

To establish a connection with DBT Core, follow these steps:

In the OvalEdge application, click on the "Administration" module from the left panel menu.
Select "Connectors", which will take users to the Connectors page.
Click on the "+" (New Connector) button enabled at the top right corner of the page. This will open the ‘Add Connector’ pop-up window.
To find the DBT Core connector, utilize the provided search bar and click on the widget associated with the DBT connector. This action will open a pop-up window labeled "Manage Connection".
Proceed to fill in the required fields with the relevant information to configure the DBT Core connection.

Note: The asterisk (*) denotes mandatory fields required for establishing a connection.

The following are the field attributes required for the connection of DBT Core.

Field Name	Description
Connector Type	It allows users to select the connector from the drop-down menu. By default, 'DBT Core' is displayed as the selected connection type.
Credential Manager*	Select the option from the drop-down list to indicate where you want to save your credentials: OE Credential Manager: DBT Core connection is configured with the basic Username and Password of the service account in real-time when OvalEdge establishes a connection to the DBT Core. Users need to add the credentials manually if the OE Credential Manager option is selected. HashiCorp: The credentials are stored in the HashiCorp database server and fetched from HashiCorp to OvalEdge. AWS Secrets Manager: The credentials are stored in the AWS Secrets Manager database server; OvalEdge fetches the credentials from the AWS Secrets Manager. Azure Key Vault: Azure Key Vault allows for secure storage and strict access mechanisms of sensitive information such as tokens, passwords, certificates, API keys, and other confidential data. For more information on Azure Key Vault, refer to Azure Key Vault For more information on Credential Manager, refer to Credential Manager
License Add Ons	OvalEdge offers a default Base Connector License for all connectors, allowing users to crawl and profile data sources to obtain metadata and statistical information. Additionally, OvalEdge provides various License Add-Ons to cater to different connector functionality requirements: Auto Lineage Add-On: To enable the automatic construction of data object Lineage for a connector, choose the Auto Lineage Add-On license, which supports the Lineage feature.
Credential Manager ConnId	When you have more than one Credential Manager ID, pick the specific ID you want in the Credential Manager ConnId field.
Connector Environment	Select the desired environment for the connector from the dropdown list. The environment field provides information about the specific environment in which the connector is established such as PROD, or STG. Note: The steps to set up environment variables are explained in the prerequisite section.
Connector Name*	Provide a connector name for the DBT Core database in OvalEdge. This name will serve as a reference to identify the specific DBT Core database connection. Example: "DBTCore_Connection_test"
DBT Core Repository Type*	DBT Core Repository type: Choose the appropriate option from the drop-down list. NFS S3 If you select S3 from the drop-down list, enter the following Authentication: IAM User Authentication verifies the identity of individual users attempting to access a system or resource. Role-based Authentication is not supported in DBT Core for S3. S3 Bucket name: Enter the name of the S3 bucket where DBT resides. S3 zip pathname name: Enter the path name where the zip file got extracted from the manifest file from the project folder. Access key*: Enter the Access Key of the IAM user. Secret key*: Enter the Secret Key of the IAM user. Filter by tags: Enter the tags associated with the Bucket/Object, if any. Region: Region of S3. SSO Connection Id: Connection Id of the identity provider’s connection [Azure, Okta, AVM … etc] SSO Application Id: Application Id crawled from the identity provider’s connection [Azure, Okta, AVM … etc] SSO Role Prefix: Role name from the crawled roles of the identity provider’s connection [Azure, Okta, AVM … etc] RDAM Policy Folder Path**: Bucket/Folder path in the S3 to write the policies.
Path*	If the ‘NFS’ type is selected, then the path of DBT Projects is mandatory.
Default Governance Roles*	Steward*: Select the Steward from the drop-down options.
	Custodian*: Select the Custodian from the drop-down options.
	Owner*: Select the Owner from the drop-down options.
Admin Roles*	By selecting the appropriate admin roles, users can ensure that the users associated with the selected role(s) have the required permissions and responsibilities to manage the connector effectively while maintaining security and governance. To assign the necessary admin roles for this connector, follow these steps: Integration Admins: To add Integration Admin Roles, search for or select one or more roles from the drop-down options. Once selected, click on the Apply button. Integration Admins are responsible for configuring crawling and profiling settings for the connector, as well as deleting connectors, schemas, or data objects. Security & Governance Admins: To add Security and Governance Admin roles, search for or select one or more roles from the drop-down options. Once selected, click on the Apply button. Security and Governance Admins have the following responsibilities: Configuring role permissions for the connector and its associated data objects. Adding Security and Governance Admins to set permissions for roles on the connector and its associated data objects. Updating governance roles. Creating custom attributes. Creating Service Request templates for the connector.
No. of Archive Objects*	This number represents the most recent modifications made to the metadata data of a remote/source (for report connection purposes) and can be set during the crawling connection setup.
Select Bridge*	The OvalEdge Bridge component enables seamless connectivity between cloud-hosted servers and on-premise or public cloud data sources without requiring modifications to firewall rules. It offers real-time control, simplifying the management of data movement between different sources and destinations. For more information, refer to Bridge Overview

After filling in all the AWS Secrets Manager connection details, select the appropriate button based on users preferences.

1. Validate: Click on the Validate button to verify the connection details. This ensures that the provided information is accurate and enables successful connection establishment.
2. Save: Click on the Save button to store the connection details. Once saved, the connection will be added to the Connectors home page for easy access.
3. Save & Configure: For certain connectors that require additional configuration settings. Click on the Save & Configure button. This will open the Connection Settings pop-up window, allowing you to configure the necessary settings before saving the connection.
Once the connection is validated and saved, it will be displayed on the Connectors home page.
Note: It is up to the user's wish, you can save the connection details first, or you can validate the connection first and then save it.

Error Validations Details

The following are expected errors that can be encountered while establishing the connection.

S.No.

Error Messages

Description

Failed to establish a connection, Please check the credentials

Provide valid credentials or ensure proper access.

Error occurred while validating DBT Core connection.

403: Access denied [Provide appropriate access to user or role using in connection]

404: No such key [The object does not exist in the remote.]

Note: If you have any issues creating a connection, please contact your assigned OvalEdge Customer Success Management (CSM) team.

Connector Settings

Configure the DBT Core Connector settings for crawling by following these steps:

Select the DBT Core Connection Name from the Crawler Information page.
Click on the 9 dots buttons and choose "Settings".
A "Connection Settings" pop-up window will appear for further configuration.

Configure the Lineage settings as follows:

The purpose of the lineage setting is to serve the option of changing the server/source connection to build the lineage. You can configure multiple servers simultaneously in the Select server dialects to use for parsing source codes which is mandatory and also set the Connector priority to specify the connection for table Lineage to pick the source table for lineage building.

Additional Information

NFS

When a user chooses the DBT Core Repository type as 'NFS', it is mandatory to input the NFS path.
Folder Structure: The parent folder (dbt_artifactory) should only contain DBT folders.

S3

For S3, the Bucket name should not contain any slash (/ or \). Just a bucket name is sufficient.
The Manifest.json file must be present in the target folder.

FAQ’s

Why DBT Core does not have column lineage:

The DBT API does not support column-level lineage.
As a result, OvalEdge does not have access to parse and display column lineages for source tables.