Kafka Connector

Connectivity Summary

The Kafka Connector is used to pull the metadata existing in the Kafka database and helps the users to crawl the metadata and profile the sample data and build lineage to view the movement of the crawled data.

connectors Summary-1

The drivers used by the connector are given below:

Driver/API: No external driver required.                

Version: 5.3.0

Internal jars Details: 

org.apache.kafka.clients

io.confluent.kafka-schema-registry-client

Technical Specifications

The connector capabilities are shown below:

Crawling

The crawler supports below objects:

Feature Supported Objects Remarks
Crawling Tables Topics in Kafka will be represented as tables
Table Columns

Messages in Kafka will be represented as Columns and they will be crawled while sample profiling.

Views  
Stored Procedures  
Functions  
Roles  
Users  
Permissions  
Usage Statistics  

Profiling

Feature Support Remarks
Table Profiling Row Count, Columns Count, View Sample Data  
Views Profiling Not Supported  
Column Profiling

Min, Max, Null count, Distinct, Top 50 Values

 
Full Profiling Not Supported  
Sample Profiling Supported  

For more information on profiling, please refer to the Profiling Data article.

Lineage Building

Lineage Entities Details
Table Lineage Not Supported
Column Lineage Not Supported
Lineage Sources Not Supported

Querying

Operation  Details
Select Not Supported
Insert Not Supported
Update Not Supported
Delete Not Supported
Joins within database Not Supported
Joins outside database Not Supported
Aggregations Not Supported
Group By Not Supported
Order By Not Supported

Note: As of now we haven't provided the Querying for Kafka, in future releases we are planning to implement querying by using supporting libraries.

Pre-requisites

To use the connector, the following need to be available:

  1. Connection details as specified in the following section should be available.
  2. Need a service account user with read privileges.

Connection Details

To connect to the Kafka database using the OvalEdge application, complete the following steps.

  1. Login to the OvalEdge application
  2. In the left menu, click on the Administration module name, the sub-modules associated with the Administration are displayed.
  3. Click on the Crawler sub-module name, the Crawler Information page is displayed.
  4. In the Crawler Information page, click on the . The Manage Connection with Search Connector pop-up window is displayed.
  5. In the Manage Connection pop-up window, select the connection type as Kafka. The Manage Connection with Kafka specific details pop-up window is displayed.
    Kafka Connector
  6. The following are the field attributes required for the connection of Kafka.
    Field Name

    Mandatory/Optional

    Description
    Connection Type Mandatory By default the connection type is displayed as the Kafka if needed the connection type can be changed by selecting desired connection type from the Connection Type dropdown and based on the selection of connection type, the fields associated with the selected connection type are displayed.
    License Type Mandatory Select the license type used for the connection, the available options are Standard or Auto Lineage
    Name Mandatory

    Enter the name of the connection, the connection name specified in the Connection Name textbox will be a reference to the Kafka database connection in the OvalEdge application.

    Example: Kafka Connection1

    Broker URL Mandatory

    Enter the database instance URL (on-premises/cloud-based)
    Example: oval-Kafka.csklygkwz3dx.us-east-1.rds.amazonaws.com

    Cluster Name Mandatory Enter the cluster name associated with the database the default cluster is represented as 3306. 
    JAAS Config Path Optional

    Enter the JAAS configuration path associated with the database. 

    KRB5 Config Path Optional

    Enter the KRB5 configuration path associated with the database. 

    Security Protocol Optional

    Enter security protocol details in the provided textbox.

    SASL Mechanism Optional

    Enter SASL mechanism details in the provided textbox.

    Registry URL Optional

    Enter the Registry URL details in the provided textbox.

    Example: http://3.21.242.139:8081

    Schema Registry User Optional

    Enter the schema registry username in the provided textbox.

    Schema Registry Password Optional

    Enter the schema registry password in the provided textbox.

    Once after entering the connection details in the required fields, click on the button the entered connection details are validated the Save and Save & Configure buttons are enabled.
  7. Click on the button to establish the connection or the user can also directly click on the button to establish the connection and configure the connection settings. Here when you click on the Save & Configure button, the Connection Settings pop-up window is displayed. Where you can configure the connection settings for the selected Connector.
    Note: The Save & Configure button is displayed only for the Connectors for which the settings configuration is required.
  8. Once the connectivity is established successfully the additional configurations for crawling and profiling can be specified.
  9. To configure the Crawler and Profiler settings for the Kafka Connector, select the Kafka Connection Name from the Crawler Information page and click on the 9 dots buttons and select the Settings options. The Connection Settings pop-up window is displayed. 
    Kafka Connectorsettings1

    Crawler Settings 

    Property Details
    Tables, Views & Columns By default, it is selected as True
    Crawler Rules

    By default include tables regex has  regex = (.*) , which can crawl all the tables

     Profiler Settings 

    Kafka Connectorsettings2
    Property Details
    Tables and Columns By default, it is selected as True
    Views and Columns By default, it is selected as True
    Profile Rules By default include tables regex has  regex = (.*) , which can profile all the tables

Copyright © 2019, OvalEdge LLC, Peachtree Corners GA USA.