Schema Registry

Schema Registry provides a centralized repository for managing and validating schemas for topic message data, as well as serializing and deserializing it over the network. As schemas evolve, producers and consumers of Kafka topics can use schemas to ensure data consistency and compatibility.

OvalEdge, on the other hand, offers a user-friendly interface that enables connectivity to Schema Registry and supports a wide range of standard operations. It integrates with Schema Registry using the JDBC driver, which allows for tasks such as crawling database objects, profiling sample data, executing queries, building lineage, implementing remote policies, and managing access permissions.

Prerequisites

The following are prerequisites for connecting to the Schema Registry:

The APIs/ drivers used by the connector are given below:

User Permission

An admin/service account for crawling and building lineage. The minimum privileges required are:

Configuring Environment Variables (Optional)

Configuring environment names enables you to select the appropriate environment from the drop-down list when adding a connector. This allows for consistent crawling of schemas across different environments, such as production (PROD), staging (STG), or temporary environments. It also facilitates schema comparisons and assists in application upgrades by providing a temporary environment that can be later deleted if needed.

Before establishing a connection, it is important to configure the environment names for the specific connector. If your environments have been configured, skip this step. 

1. Go to Administration > System Settings.
2. Select the Connector tab and find the Key name "connector.environment".
3. Enter the desired environment values (e.g., PROD, STG) in the value column.
4. Click on the checkmark ✔ icon to save the configuration.

Technical Specification

The connector capabilities are shown below:

Crawling

Profiling

Lineage Building

Querying

By default, the service account provided for the connector will be used for any query operations. If the service account has to write privileges, then Insert / Update / Delete queries can be executed.

Connection Details

To connect to the Schema Registry using the OvalEdge application, complete the following steps:

1. Log in to the OvalEdge application.
2. Navigate to Administration >  Connectors.
3. Click on the + (New Connector) icon, and the Add Connection with Search Connector pop-up window is displayed.
4. Add Connector pop-up window is displayed where you can search for the Schema Registry connector.
   
   
5. The Add Connector with Connector Type specific details pop-up window is displayed. Enter the relevant information to configure the Schema Registry connection.
   Note: The asterisk (*) denotes mandatory fields required for establishing a connection.
   
   

   Field Name		Description
   Connector Type		It allows you to select the connector from the drop-down list. By default, 'Schema Registry' is displayed as the selected connector type.
   Connector Settings
   License Add Ons		All the connectors will have a Base Connector License by default that allows you to crawl and profile to obtain the metadata and statistical information from a datasource.  OvalEdge supports various License Add-Ons based on the connector’s functionality requirements. Auto Lineage: Select the Auto Lineage Add-On license that enables the automatic construction of the Lineage of data objects for a connector with the Lineage feature. Data Quality: Select the Data Quality Add-On license to identify, report, and resolve the data quality issues for a connector whose data supports data quality, using DQ Rules/functions, Anomaly detection, Reports, and more. Data Access: Select the Data Access Add-On license that will enforce connector access via OvalEdge with Remote Data Access Management (RDAM) feature-enabled.
   Connector Name*		Provide a connector name for the Schema Registry database in OvalEdge. This name will serve as a reference to identify the specific Schema Registry database connection. For example, you can use "SchemaRegistry_Connection_test" as the connector name.
   Connector Environment		The Connector Environment drop-down list allows you to select the environment configured for the connector from the drop-down list.  For example, PROD, or STG (based on the configured items in the OvalEdge configuration for the connector.environment). The purpose of the environment field is to help you identify which connector is connecting what type of system environment (Production, STG, or QA). Note: The steps to set up environment variables are explained in the prerequisite section.
   Schema Registry Endpoint*
   Cloud API Key*
   Cloud API Secret*
   Default Governance Roles
   Steward*		Select the Steward from the drop-down options.
   Custodian*		Select the Custodian from the drop-down options.
   Owner*		Select the Owner from the drop-down options.
   Governance Roles 4, 5, 6*		Select the respective user from the drop-down options. Note: The drop-down list displays all the configurable roles (single user or a team) as per the configurations made in the OvalEdge Security > Governance Roles section.
   Admin Roles
   Integration Admins*		To add Integration Admin Roles, search for or select one or more roles from the Integration Admin options, and then click on the Apply button. The responsibility of the Integration Admin includes configuring crawling and profiling settings for the connector, as well as deleting connectors, schemas, or data objects.
   Security and Governance Admins*		To add Security and Governance Admin roles, search for or select one or more roles from the list, and then click on the Apply button. The security and Governance Admin is responsible for: Configure role permissions for the connector and its associated data objects. Add admins to set permissions for roles on the connector and its associated data objects. Update governance roles. Create custom fields. Develop Service Request templates for the connector. Create Approval workflows for the templates.
   Select Bridge		With the OvalEdge Bridge component, any cloud-hosted server can connect with any on-premise or public cloud data sources without modifying firewall rules. A bridge provides real-time control that makes it easy to manage data movement between any source and destination. For more information, refer to Bridge Overview. For more information, refer to Bridge Overview

6. After filling in all the connection details, select the appropriate button based on your preferences. 
   1. Validate: Click on the Validate button to verify the connection details. This ensures that the provided information is accurate and enables successful connection establishment.
   2. Save: Click on the Save button to store the connection details. Once saved, the connection will be added to the Connectors home page for easy access.
   3. Save & Configure: For certain Connectors that require additional configuration settings. Click on the Save & Configure button. This will open the Connection Settings pop-up window, allowing you to configure the necessary settings before saving the connection.
7. Once the connection is validated and saved, it will be displayed on the Connectors home page.

Connection Settings

Crawler

Profiler