The Kafka Connector is used to pull the metadata existing in the Kafka database and helps the users to crawl the metadata and profile the sample data and build lineage to view the movement of the crawled data.
The drivers used by the connector are given below:
Driver/API: No external driver required.
Internal jars Details:
The connector capabilities are shown below:
The crawler supports below objects:
|Crawling||Tables||Topics in Kafka will be represented as tables|
Messages in Kafka will be represented as Columns and they will be crawled while sample profiling.
|Table Profiling||Row Count, Columns Count, View Sample Data|
|Views Profiling||Not Supported|
Min, Max, Null count, Distinct, Top 50 Values
|Full Profiling||Not Supported|
For more information on profiling, please refer to the Profiling Data article.
|Table Lineage||Not Supported|
|Column Lineage||Not Supported|
|Lineage Sources||Not Supported|
|Joins within database||Not Supported|
|Joins outside database||Not Supported|
|Group By||Not Supported|
|Order By||Not Supported|
Note: As of now we haven't provided the Querying for Kafka, in future releases we are planning to implement querying by using supporting libraries.
To use the connector, the following need to be available:
- Connection details as specified in the following section should be available.
- Need a service account user with read privileges.
To connect to the Kafka database using the OvalEdge application, complete the following steps.
- Login to the OvalEdge application
- In the left menu, click on the Administration module name, the sub-modules associated with the Administration are displayed.
- Click on the Crawler sub-module name, the Crawler Information page is displayed.
- In the Crawler Information page, click on the . The Manage Connection with Search Connector pop-up window is displayed.
- In the Manage Connection pop-up window, select the connection type as Kafka. The Manage Connection with Kafka specific details pop-up window is displayed.
- The following are the field attributes required for the connection of Kafka.
Description Connection Type Mandatory By default the connection type is displayed as the Kafka if needed the connection type can be changed by selecting desired connection type from the Connection Type dropdown and based on the selection of connection type, the fields associated with the selected connection type are displayed. License Type Mandatory Select the license type used for the connection, the available options are Standard or Auto Lineage Name Mandatory
Enter the name of the connection, the connection name specified in the Connection Name textbox will be a reference to the Kafka database connection in the OvalEdge application.
Example: Kafka Connection1
Broker URL Mandatory
Enter the database instance URL (on-premises/cloud-based)
Cluster Name Mandatory Enter the cluster name associated with the database the default cluster is represented as 3306. JAAS Config Path Optional
Enter the JAAS configuration path associated with the database.
KRB5 Config Path Optional
Enter the KRB5 configuration path associated with the database.
Security Protocol Optional
Enter security protocol details in the provided textbox.
SASL Mechanism Optional
Enter SASL mechanism details in the provided textbox.
Registry URL Optional
Enter the Registry URL details in the provided textbox.
Schema Registry User Optional
Enter the schema registry username in the provided textbox.
Schema Registry Password Optional
Enter the schema registry password in the provided textbox.
- Click on the button to establish the connection or the user can also directly click on the button to establish the connection and configure the connection settings. Here when you click on the Save & Configure button, the Connection Settings pop-up window is displayed. Where you can configure the connection settings for the selected Connector.
Note: The Save & Configure button is displayed only for the Connectors for which the settings configuration is required.
- Once the connectivity is established successfully the additional configurations for crawling and profiling can be specified.
- To configure the Crawler and Profiler settings for the Kafka Connector, select the Kafka Connection Name from the Crawler Information page and click on the 9 dots buttons and select the Settings options. The Connection Settings pop-up window is displayed.
Property Details Tables, Views & Columns By default, it is selected as True Crawler Rules
By default include tables regex has regex = (.*) , which can crawl all the tables
Property Details Tables and Columns By default, it is selected as True Views and Columns By default, it is selected as True Profile Rules By default include tables regex has regex = (.*) , which can profile all the tables
Copyright © 2019, OvalEdge LLC, Peachtree Corners GA USA.