Connectivity Summary
The Kafka Connector is used to pull the metadata existing in the Kafka database and helps the users to crawl the metadata and profile the sample data and build lineage to view the movement of the crawled data.
The drivers used by the connector are given below:
Driver/API: No external driver required.
Version: 5.3.0
Internal jars Details:
org.apache.kafka.clients
io.confluent.kafka-schema-registry-client
Technical Specifications
The connector capabilities are shown below:
Crawling
The crawler supports below objects:
Feature | Supported Objects | Remarks |
Crawling | Tables | Topics in Kafka will be represented as tables |
Table Columns |
Messages in Kafka will be represented as Columns and they will be crawled while sample profiling. |
|
Views | ||
Stored Procedures | ||
Functions | ||
Roles | ||
Users | ||
Permissions | ||
Usage Statistics |
Profiling
Feature | Support | Remarks |
Table Profiling | Row Count, Columns Count, View Sample Data | |
Views Profiling | Not Supported | |
Column Profiling |
Min, Max, Null count, Distinct, Top 50 Values |
|
Full Profiling | Not Supported | |
Sample Profiling | Supported |
For more information on profiling, please refer to the Profiling Data article.
Lineage Building
Lineage Entities | Details |
Table Lineage | Not Supported |
Column Lineage | Not Supported |
Lineage Sources | Not Supported |
Querying
Operation | Details |
Select | Not Supported |
Insert | Not Supported |
Update | Not Supported |
Delete | Not Supported |
Joins within database | Not Supported |
Joins outside database | Not Supported |
Aggregations | Not Supported |
Group By | Not Supported |
Order By | Not Supported |
Note: As of now we haven't provided the Querying for Kafka, in future releases we are planning to implement querying by using supporting libraries.
Pre-requisites
To use the connector, the following need to be available:
- Connection details as specified in the following section should be available.
- Need a service account user with read privileges.
Connection Details
To connect to the Kafka database using the OvalEdge application, complete the following steps.
- Login to the OvalEdge application
- In the left menu, click on the Administration module name, the sub-modules associated with the Administration are displayed.
- Click on the Crawler sub-module name, the Crawler Information page is displayed.
- In the Crawler Information page, click on the
. The Manage Connection with Search Connector pop-up window is displayed.
- In the Manage Connection pop-up window, select the connection type as Kafka. The Manage Connection with Kafka specific details pop-up window is displayed.
- The following are the field attributes required for the connection of Kafka.
Field Name Mandatory/Optional
Description Connection Type Mandatory By default the connection type is displayed as the Kafka if needed the connection type can be changed by selecting desired connection type from the Connection Type dropdown and based on the selection of connection type, the fields associated with the selected connection type are displayed. License Type Mandatory Select the license type used for the connection, the available options are Standard or Auto Lineage Name Mandatory Enter the name of the connection, the connection name specified in the Connection Name textbox will be a reference to the Kafka database connection in the OvalEdge application.
Example: Kafka Connection1
Broker URL Mandatory Enter the database instance URL (on-premises/cloud-based)
Example: oval-Kafka.csklygkwz3dx.us-east-1.rds.amazonaws.comCluster Name Mandatory Enter the cluster name associated with the database the default cluster is represented as 3306. JAAS Config Path Optional Enter the JAAS configuration path associated with the database.
KRB5 Config Path Optional Enter the KRB5 configuration path associated with the database.
Security Protocol Optional Enter security protocol details in the provided textbox.
SASL Mechanism Optional Enter SASL mechanism details in the provided textbox.
Registry URL Optional Enter the Registry URL details in the provided textbox.
Example: http://3.21.242.139:8081
Schema Registry User Optional Enter the schema registry username in the provided textbox.
Schema Registry Password Optional Enter the schema registry password in the provided textbox.
button the entered connection details are validated the Save and Save & Configure buttons are enabled.
- Click on the
button to establish the connection or the user can also directly click on the
button to establish the connection and configure the connection settings. Here when you click on the Save & Configure button, the Connection Settings pop-up window is displayed. Where you can configure the connection settings for the selected Connector.
Note: The Save & Configure button is displayed only for the Connectors for which the settings configuration is required. - Once the connectivity is established successfully the additional configurations for crawling and profiling can be specified.
- To configure the Crawler and Profiler settings for the Kafka Connector, select the Kafka Connection Name from the Crawler Information page and click on the 9 dots buttons and select the Settings options. The Connection Settings pop-up window is displayed.
Crawler Settings
Property Details Tables, Views & Columns By default, it is selected as True Crawler Rules By default include tables regex has regex = (.*) , which can crawl all the tables
Profiler Settings
Property Details Tables and Columns By default, it is selected as True Views and Columns By default, it is selected as True Profile Rules By default include tables regex has regex = (.*) , which can profile all the tables
Copyright © 2019, OvalEdge LLC, Peachtree Corners GA USA.