ETLs
      Back to home
      1. Knowledge Base
      2. Connectors
      3. ETLs

      Talend Connector

      Talend is a comprehensive collection of services and software solutions for managing data from multiple sources. Talend's data integration tools make it easy for businesses to quickly combine data from various sources, such as databases, flat files, online services, and web API Management.

      Currently, OvalEdge uses Azure DevOps Repo connector ID to connect to the Talend data source, which allows the user to crawl Workflows into the OvalEdge application and build Lineage.

      Connector Capabilities

      The following is the list of objects supported by the Talend.

      Functionality

      Supported Objects

      Crawling 

      Workflows

      Prerequisites

      The following are the prerequisites required for establishing a connection between the connector and the OvalEdge application.

      1. File System connection ID
      2. Configure environment variables (Optional)

      File System connection ID

      Currently, OvalEdge uses Azure DevOps Repo connector ID to connect to the Talend data source. By using this ID in the Talend connector’s Workspace Connection ID field, we establish a connection into OvalEdge.

      Configure Environment Variables (Optional)

      This section describes the settings or instructions you should know before establishing a connection. If your environments have been configured, skip this step.

      Configure Environment Names

      The Environment Names allow you to select the environment configured for the specific connector from the dropdown list in the Add Connector pop-up window.

      You might want to consider crawling the same schema in both stage and production environments for consistency. The typical environments for crawling are PROD, STG, or Temporary, and may also include QA or other environments. Additionally, crawling a temporary environment can be useful for schema comparisons, which can later be deleted, especially during application upgrade assistance. 

      Steps to Configure the Environment

      1. Navigate to Administration > System Settings.
      2. Select the Connector tab.
      3. Find the Key name “connector.environment”.
      4. Enter the desired environment values (PROD, STG) in the value column. 
      5. Click ✔ to save. 

      Service Account with Minimum Permissions

      A Service Account with Minimum Permissions relies on another type of file connector and its permissions. Currently, OvalEdge has only been set up for the Azure DevOps Repo file system.

      Establish a Connection 

      To establish a connection, complete the following steps:

      1. Log in to the OvalEdge application.
      2. Navigate to Administration >  Connectors.
      3. Click on the + (New Connector) icon.
      4. Add Connector pop-up window is displayed where you can search for the Talend connector.
      5. The Add Connector with Connector Type specific details pop-up window is displayed. Enter the relevant information to configure the Talend connection.
        Note: The asterisk (*) denotes mandatory fields required for establishing a connection.

         

        Field Name

        Description

        Connector Type

        By default, the selected connection type is displayed as the Talend. 

        If required, the drop-down list allows the user to change the connector type and based on the selection of the connection type, the fields associated with the selected connection type are displayed.

        Credential Manager*

        The purpose of a credential manager is to enhance the security that stores the API keys, passwords, certificates, and other sensitive data securely and helps to manage access, rotates, and audit secrets. 

        OE Credential Manager: Talend connection is configured with the basic Username and Password of the service account in real-time when OvalEdge establishes a connection to the Talend database. Users need to add the credentials manually if the OE Credential Manager option is selected.

        HashiCorp: The credentials are stored in the HashiCorp database server and fetched from HashiCorp to OvalEdge.

        AWS Secrets Manager: The credentials are stored in the AWS Secrets Manager database server and fetched from the AWS Secrets Manager to OvalEdge.

        Azure Key Vault: Refer to Azure Key Vault.

        For more information on Credential Manager, refer to Credential Manager

        License Add-Ons

        All the connectors will have a Base Connector License by default that allows you to crawl and profile to obtain the metadata and statistical information from a data source.

        OvalEdge supports various License Add-Ons based on the connector’s functionality requirements.

        • Select the Auto Lineage Add-On license that enables the automatic construction of the Lineage of data objects for a connector with the Lineage feature. 

        Connector Name*

        Select a connection name for Talend. You can specify a connection name to identify the Talend connection in OvalEdge.

        Example: Talend_test

        Connector Environment

        The environment drop-down list allows you to select the environment configured for the connector from the drop-down list. For example, PROD, or STG (based on the configured items in the OvalEdge configuration for the connector.environment).

        The purpose of the environment field is to help you identify which connector is connecting what type of system environment (Production, STG, or QA).  

        Note: The steps to set up environment variables are explained in the prerequisites section.

        Admin Roles*

        Select the required admin roles for this connector.

        • To add Integration Admin Roles, search for or select one or more roles from the Integration Admin options, and then click on the Apply button.
          The responsibility of the Integration Admin includes configuring crawling and profiling settings for the connector, as well as deleting connectors, schemas, or data objects.
        • To add Security and Governance Admin roles, search for or select one or more roles from the list, and then click on the Apply button.
          The security and Governance Admin is responsible for:
          • Configure role permissions for the connector and its associated data objects.
          • Add admins to set permissions for roles on the connector and its associated data objects.
          • Update governance roles.
          • Create custom fields.
          • Develop Service Request templates for the connector.
          • Create Approval workflows for the templates.

        Workspace Connection Id*

        File connection id.

        For more information, click here.

        contextParams File Path

        If you're using Global variables while creating workflows, you can use the Excel file to define and specify those variables. This is used in building the lineage.

        Select Bridge*

        A solution is required to circumnavigate the customer firewall when OvalEdge is deployed as a SaaS application. That solution is OvalEdge Bridge. A bridge is a type of firewall that operates at the network layer. 

        • When a bridge has been set up, it will be displayed here in a dropdown menu. Users can select the required Bridge ID.
        • The user can select "NO BRIDGE" when it is not configured.

        For more information, refer to Bridge Overview
        Note: It is up to the user's wish, you can save the connection details first, or you can validate the connection first and then save it.

      6. After entering all the connection details, select the appropriate button based on your preferences.
        1. Validate: Click on the Validate button to verify the connection details. This ensures that the provided information is accurate and enables successful connection establishment.
        2. Save: Click on the Save button to store the connection details. Once saved, the connection will be added to the Connectors home page for easy access.
        3. Save & Configure: For certain Connectors that require additional configuration settings. Click on the Save & Configure button. This will open the Connection Settings pop-up window, allowing you to configure the necessary settings before saving the connection.
      7. Once the connection is validated and saved, it will be displayed on the Connectors home page.

        Note: You can either save the connection details first, or you can validate the connection first and then save it.

      Connection Validation Errors

      Error Messages

      Description

      Failed to establish a connection, please check the connection id.

      In case of an invalid connection id.

      Note: If you have any issues creating a connection, please contact your assigned OvalEdge Customer Success Management (CSM) team.

      Connection Settings

      Lineage

      For the Talend Connector, only the Lineage setting option is available, and it is enabled when the Auto Lineage License option is selected.

      The purpose of the lineage setting is to serve the option of changing the server/source connection to build the lineage. You can configure multiple servers simultaneously in the  Selecting Source Server Type for lineage and also set the connection priority lists to pick the tables in the Connections Priority to pick the source table for lineage building.

      The Crawling of Schema(s)

      You can use the Crawl/Profile option, which allows you to select the specific schemas for the following operations: crawl, profile, crawl & profile, or profile unprofiled. For any scheduled crawlers and profilers, the defined run date and time are displayed to set.

      1. Navigate to the Connectors page, and click on the Crawl/Profile option.
      2. Select the required Schema(s).
      3. Click on the Run button that gathers all metadata from the connected source into the OvalEdge Data Catalog.

      Note: For more information on Scheduling, refer to Scheduling Connector.