ETLs
      Back to home
      1. Knowledge Base
      2. Connectors
      3. ETLs

      AWS Database Migration Service

      AWS Database Migration Service is a cloud-based solution designed to migrate various types of data stores, such as relational databases, data warehouses, and NoSQL databases. It enables seamless data migration between the AWS Cloud and hybrid cloud/on-premises environments.

      OvalEdge uses a Java SDK driver to connect to the data source, which allows users to crawl and build lineage.

      0

      Connector Capabilities

      The connector capabilities are shown below:

      Crawling

      Features

      Supported Objects

      Remarks

      Crawling of Metadata Objects

      Tasks

      -

      Prerequisites

      The following are the prerequisites for connecting the connector and the OvalEdge application.

      Driver Details

      Sl.No

      Driver / API

      Version

      Details

      1

      SDK Driver

      1.12.661

      https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk-dms/1.12.661

      Service User Account and Permissions

      To crawl metadata into OvalEdge, create an AWS Database Migration Service account with the following permissions: Create an AWS Database Migration Service account to connect to OvalEdge. The required permissions and corresponding AWS Database Migration Service commands are given below for your reference.

      Operation 

      Minimum Access Permission 

      Connection Validation

      Read

      Crawl Database

      Read

      Crawl ETL Sourcecode

      Read

      Lineage

      Read

      Configuring Environment Variables

      Configuring environment names enables you to select the appropriate environment from the drop-down list when adding a connector. This allows for consistent crawling of schemas across different environments, such as production (PROD), staging (STG), or temporary environments. It also facilitates schema comparisons and assists in application upgrades by providing a temporary environment that can later be deleted.

      Before establishing a connection, it is important to configure the environment names for the specific connector. If your environments have been configured, skip this step. 

      Steps to Configure the Environment

      1. Log into the OvalEdge application.
      2. Navigate to Administration >  System Settings.
      3. Select the Connector tab.
      4. Find the key name “connector.environment”.
        1
      5. Enter the desired environment values (PROD, STG) in the Value column.
      6. Click ✔ to Save.

      Establish a Connection

      To connect to AWS Database Migration Service using the OvalEdge application, complete the following steps:

      1. Log in to the OvalEdge application.
      2. Navigate to Administration >  Connectors.
      3. Click on the + (New Connector) icon.
        2
      4. The Add Connector pop-up window is displayed, and you can search for the AWS Database Migration Service connector.
        3
      5. The Add Connector with Connector Type specific details pop-up window is displayed. Enter the relevant information to configure the AWS Database Migration Service connection.
        Note: An asterisk (*) denotes a mandatory field for establishing a connection.

        Field Name

        Description

        Connector Type

        This field allows you to select the connector from the drop-down list provided. By default, 'AWS Database Migration Service' is displayed as the selected connector type.

        Authentication*

        IAM User Authentication or Role-Based Authentication.

        Credential Manager*

        Select the option from the drop-down menu where you want to save your credentials:


        OE Credential Manager: When OvalEdge establishes a connection to the AWS Database Migration Service database, it is configured with the service account's basic username and password. Users must manually add the credentials if the OE Credential Manager option is selected.

        HashiCorp: The credentials are stored in the HashiCorp database server and fetched from HashiCorp to OvalEdge.

        AWS Secrets Manager: The credentials are stored in the AWS Secrets Manager database server and fetched from the AWS Secrets Manager to OvalEdge.

        For more information on Azure Key Vault, refer to Azure Key Vault.

        For more information on Credential Manager, refer to Credential Manager.

        License Add Ons

        By default, all the connectors will have a Base Connector License, which allows you to crawl and profile to obtain metadata and statistical information from a data source.

         OvalEdge supports various License Add-Ons based on the connector’s functionality requirements.

        • Select the Auto Lineage Add-On license that enables the automatic construction of the Lineage of data objects for a connector with the Lineage feature.

        Connector Name*

        The connection name refers to the AWS Database Migration Service database connection in the OvalEdge application.

        Connector Environment

        The Connector Environment drop-down list allows you to select the environment configured for the connector from the drop-down list. 

        For example, you can select PROD or STG (based on the items configured in the OvalEdge configuration for the connector environment).

        The purpose of the environment field is to help you identify which connector is connecting what type of system environment (Production, STG, or QA).

        Note: The Configuring Environment Variables section explains setting up environment variables.

        Access Key*

        An Access Key in AWS Database Migration Service is a security credential to access AWS resources.

        Secret Key*

        The Secret Key in AWS Database Migration Service is part of the security credentials to access AWS resources. It is paired with an Access Key ID.

        Database region*

        The "Database region" in AWS Database Migration Service denotes the geographic location of the source and target databases within AWS infrastructure.

        Default Governance Roles

        Steward*

        Select the Steward from the drop-down list options.

        Custodian*

        Select the Custodian from the drop-down list options.

        Owner*

        Select the Owner from the drop-down list options.

        Governance Roles 4, 5, 6*

        Select the respective user from the drop-down options.


        Note: The drop-down list displays all the configurable roles (for a single user or a team) according to the configurations made in the OvalEdge Security > Governance Roles section.

        Admin Roles

        Integration Admins*

        To add Integration Admin Roles, search for or select one or more roles from the Integration Admin options, then click the Apply button.
        The Integration Admin's responsibilities include configuring the connector's crawling and profiling settings and deleting connectors, schemas, or data objects.

        Security and Governance Admins*

        To add Security and Governance Admin roles, search for or select one or more roles from the list and click the Apply button.
        The Security and Governance Admin is responsible for:

        • Configuring role permissions for the connector and its associated data objects.
        • Adding admins to set permissions for the connector's roles and associated data objects.
        • Updating governance roles.
        • Creating custom fields.
        • Developing Service Request templates for the connector.
        • Creating approval workflows for  Service Request templates.

        Select Bridge*

        With the OvalEdge Bridge component, any cloud-hosted server can connect with any on-premise or public cloud data source(s) without modifying firewall rules. A bridge provides real-time control, making data movement between source and destination easy. For more information, refer to

        Bridge Overview.
      6. After entering all the required connection details, select the appropriate option based on your preferences:
        1. Validate: Click the Validate button to verify the connection details. This ensures that the provided information is accurate and enables successful connection establishment.
        2. Save: Click on the Save button to store the connection details. Once saved, the connection will be added to the Connectors home page for easy access.
        3. Save & Configure: For certain Connectors requiring additional configuration settings, click the Save & Configure button. This will open the Connection Settings pop-up window, allowing you to configure the necessary settings before saving the connection.
      7. Once the connection is validated and saved, it will be displayed on the Connectors home page.
        Note: You can either save the connection details first or validate the connection first and then save it.

      Connection Validation Details

      S.No

      Error Message(s)

      Description

      1

      Failed to establish a connection. Please check the credentials.

      Invalid credentials are provided, or the user or role does not have access.

      2

      Connection Timeout

      Invalid credentials are provided, or the server is not running.

      Note: If you have issues creating a connection, please contact your assigned OvalEdge Customer Success Management (CSM) team.

      Connector Settings

      To view the Connector Settings page,

      1. Go to the Connectors page.
      2. From the 9- dots, select the Settings option.
        5
      3. This will display the Connector Settings page, where you can view the connector settings.
        6
      Note: For more information, refer to the Connector Settings.

      Crawling of Schema(s)

      You can use the Crawl/Profile option, which allows you to select the specific schemas for the following operations: crawl, profile, crawl & profile, or profile unprofiled. The defined run date and time are displayed to set for any scheduled crawlers and profilers.
      1. Navigate to the Connectors page and click the Crawl/Profile option.
      2. Select the required Schema(s).
      3. Click on the Run button, which gathers all metadata from the connected source and puts it into the OvalEdge Data Catalog.

      Note: For more information on Scheduling, refer to Scheduling Connector.