Kafka Connector

Connectivity Summary

The Kafka Connector is used to pull the metadata existing in the Kafka database and helps the users to crawl the metadata and profile the sample data and build lineage to view the movement of the crawled data.

connectors Summary-1

The drivers used by the connector are given below:

Driver/API: No external driver required.

Version: 5.3.0

Internal jars Details:

org.apache.kafka.clients

io.confluent.kafka-schema-registry-client

Technical Specifications

The connector capabilities are shown below:

Crawling

The crawler supports below objects:

Feature	Supported Objects	Remarks
Crawling	Tables	Topics in Kafka will be represented as tables
	Table Columns	Messages in Kafka will be represented as Columns and they will be crawled while sample profiling.
	Views
	Stored Procedures
	Functions
	Roles
	Users
	Permissions
	Usage Statistics

Profiling

Feature	Support	Remarks
Table Profiling	Row Count, Columns Count, View Sample Data
Views Profiling	Not Supported
Column Profiling	Min, Max, Null count, Distinct, Top 50 Values
Full Profiling	Not Supported
Sample Profiling	Supported

For more information on profiling, please refer to the Profiling Data article.

Lineage Building

Lineage Entities	Details
Table Lineage	Not Supported
Column Lineage	Not Supported
Lineage Sources	Not Supported

Querying

Operation	Details
Select	Not Supported
Insert	Not Supported
Update	Not Supported
Delete	Not Supported
Joins within database	Not Supported
Joins outside database	Not Supported
Aggregations	Not Supported
Group By	Not Supported
Order By	Not Supported

Note: As of now we haven't provided the Querying for Kafka, in future releases we are planning to implement querying by using supporting libraries.

Pre-requisites

To use the connector, the following need to be available:

Connection details as specified in the following section should be available.
Need a service account user with read privileges.

Connection Details

To connect to the Kafka database using the OvalEdge application, complete the following steps.

Login to the OvalEdge application
In the left menu, click on the Administration module name, the sub-modules associated with the Administration are displayed.
Click on the Crawler sub-module name, the Crawler Information page is displayed.
In the Crawler Information page, click on the . The Manage Connection with Search Connector pop-up window is displayed.
In the Manage Connection pop-up window, select the connection type as Kafka. The Manage Connection with Kafka specific details pop-up window is displayed.

The following are the field attributes required for the connection of Kafka.

Field Name	Mandatory/Optional	Description
Connection Type	Mandatory	By default the connection type is displayed as the Kafka if needed the connection type can be changed by selecting desired connection type from the Connection Type dropdown and based on the selection of connection type, the fields associated with the selected connection type are displayed.
License Type	Mandatory	Select the license type used for the connection, the available options are Standard or Auto Lineage
Name	Mandatory	Enter the name of the connection, the connection name specified in the Connection Name textbox will be a reference to the Kafka database connection in the OvalEdge application. Example: Kafka Connection1
Broker URL	Mandatory	Enter the database instance URL (on-premises/cloud-based) Example: oval-Kafka.csklygkwz3dx.us-east-1.rds.amazonaws.com
Cluster Name	Mandatory	Enter the cluster name associated with the database the default cluster is represented as 3306.
JAAS Config Path	Optional	Enter the JAAS configuration path associated with the database.
KRB5 Config Path	Optional	Enter the KRB5 configuration path associated with the database.
Security Protocol	Optional	Enter security protocol details in the provided textbox.
SASL Mechanism	Optional	Enter SASL mechanism details in the provided textbox.
Registry URL	Optional	Enter the Registry URL details in the provided textbox. Example: http://3.21.242.139:8081
Schema Registry User	Optional	Enter the schema registry username in the provided textbox.
Schema Registry Password	Optional	Enter the schema registry password in the provided textbox.

Once after entering the connection details in the required fields, click on the

button the entered connection details are validated the Save and Save & Configure buttons are enabled.

Click on the button to establish the connection or the user can also directly click on the button to establish the connection and configure the connection settings. Here when you click on the Save & Configure button, the Connection Settings pop-up window is displayed. Where you can configure the connection settings for the selected Connector.
Note: The Save & Configure button is displayed only for the Connectors for which the settings configuration is required.
Once the connectivity is established successfully the additional configurations for crawling and profiling can be specified.

To configure the Crawler and Profiler settings for the Kafka Connector, select the Kafka Connection Name from the Crawler Information page and click on the 9 dots buttons and select the Settings options. The Connection Settings pop-up window is displayed.
Kafka Connectorsettings1

Crawler Settings

Property	Details
Tables, Views & Columns	By default, it is selected as True
Crawler Rules	By default include tables regex has regex = (.*) , which can crawl all the tables

Profiler Settings

Property	Details
Tables and Columns	By default, it is selected as True
Views and Columns	By default, it is selected as True
Profile Rules	By default include tables regex has regex = (.*) , which can profile all the tables