Release6.1.2

About OvalEdge Release6.1.2

The latest OvalEdge Release6.1.2 brings substantial enhancements to various modules like Data Catalog, Business Glossary, Administration, New Connectors, and New System Settings. This release also fixed various bugs for the following modules, Business Glossary, Advanced Tools, and Administration. 

This release also resolves critical and significant bugs in various areas, ensuring a smoother experience.

Improvements 

Data Catalog

• In the Data Catalog > Tables > List view, users can select tables in bulk and click on the Nine Dots, then click on "Update Certification." In the table Summary, if certification is applied to the entire table, the certification logo is visible only at the summary tab. If users apply certification only to specific columns, they can see the certification logo only in the column details statistics tab next to the respective table column.

• In the Data Catalog > Codes > List View, to address the issue where it was challenging to differentiate between codes with similar names, a new column field "Parent Dataset'  has been added to the Configure View window. When enabled, this field includes the parent dataset's query name code in the list view improving the ability to distinguish between parent datasets,  when dealing with codes that share similar names.
• In the Data Catalog > List View,  Bulk certification was supported for Tables, Files, and Reports. Now,  the bulk updating capability has been extended to the column level for table columns, file columns, and report columns within the "Nine Dots" option in the list view.

Business Glossary 

• In the Business Glossary > Term Detailed Page > Related Objects > When linking a term to data objects or other terms, it was often difficult to determine which database or schema the data object was associated with, especially when there were similar object names.
  Now, significant enhancements have been implemented. Clicking on the '+' icon to add relations triggers a pop-up display. Within this pop-up, users can select the object type from a drop-down menu, and select the connection name, schema, and search function to add the relevant objects.
  Furthermore, the relationship type is pre-configured in the 'System Settings' while the data object is added. In the related objects list, when hovering over the list of related objects, an icon appears, allowing users to change the relationship type according to their preferences.

Advanced Tools

• In the Advanced Tools > Impact Analysis, when users attempted to discover impacted objects, the Job Status of objects was incorrectly shown as "INIT" even after the jobs had been successfully submitted. This issue has been resolved, and users can now see the job status as 'Success' upon successful job submission.
• In Advanced Tools > Load Metadata Files, users can now create external users using a DQR Excel sheet and edit existing external user details. This enhancement streamlines user management within the system.

Administration

• In the Administration > Connectors > Crawl/Profile, to enhance data profiling within OvalEdge, "Random Sampling" profiling on connectors that support this feature (Snowflake). In Simple Random Sampling, each record or data point has an equal chance of being selected for the sample. This differs from Sample Profiling, where only the top datasets are profiled, leaving the remainder unexamined.
• In the Administration > Connectors > Redshift Connector > Connector Settings, a new feature has been introduced. This enhancement enables the synchronization of OvalEdge policies with the source system within Data Access. This allows the enabling of connector policies for authorization roles and users in databases, as well as masking policies for table columns
• In the Administration > Connectors > Delta Lake Connectors, we had the option to crawl the Delta Lake connector, but there was no option to build lineage. We have enhanced the Unity Catalog lineage for the Delta Lake connector, and now users can build column-level lineage for it in most cases.
• In the Administration > Connectors > Redshift Connector > Settings > Remote Access > Remote Policy, OvalEdge is excited to introduce support for Dynamic Data Masking, powered by Redshift RDAM, enhancing user capabilities by seamlessly interacting with policies from remote sources, creating new policies within OvalEdge, and syncing them with their remote counterparts.
• In the Administration > Advanced Jobs > Extract queries from source to build lineage, the issue of duplicate queries causes performance problems and prolonged job execution times. This functionality has been enhanced for effective lineage building by comparing the queries used for lineage with those extracted from the database.

New Connectors 

• Data Pipeline Connector  

The Data Pipeline serves as a standard agreement between data producers and consumers regarding data representation format. In this context, several client services send event messages to Kafka by adhering to Data Pipeline schemas, which are structured in JSON format. Data Lake data is organized into numerous buckets and sub-buckets. These data subdivisions are grouped based on topics, named according to the current Kafka pipeline. Each topic can have varying numbers of associated schemas. Once the schemas are specified, files are further categorized based on their date and time of entry. To enhance data source visibility, OvalEdge has introduced a data pipeline connector. This connector retrieves schemas from GitHub, fetches tables and columns from JFrog, and performs data profiling using Amazon S3.

• Schema Registry 

OvalEdge uses a Schema Registry to facilitate Kafka schema management. This registry plays a crucial role in maintaining data consistency and compatibility as schemas evolve over time. It serves as a central component of data governance, contributing to data quality assurance and standards compliance. OvalEdge categorizes all schemas as tables, and each subject schema is associated with specific properties. These schema properties correspond to the columns within the schema, allowing for a structured and organized approach to schema management and data governance.

• Databricks Unity Catalog Databricks 

Unity Catalog is a leading solution for unified data and AI governance in lakehouse environments. It enables organizations to seamlessly manage data, models, notebooks, and more across cloud platforms. Users can securely access and collaborate on trusted assets, boosting productivity with AI. Unity Catalog offers centralized access control, auditing, lineage tracking, and data discovery for Azure Databricks workspaces. OvalEdge uses a JDBC driver to connect and perform data crawling to get metadata, profiling, query execution, and lineage building.

New System Settings 

The latest release introduces a new configuration that provides users with even greater control over the application's behavior. The newly added configuration is shown in the below Table.

Bug Fixes

Data Catalog

• In the Data Catalog > Tables > Table Columns > Entity Relationships, when users attempted to perform a crawl, the relationships failed to be imported into OvalEdge, even though the checkbox in the crawler settings was enabled. This issue has been resolved by removing the space after the column name using TRIM method. Users can now see the relationships when crawled.

Business Glossary

• In the Business Glossary > List View > Terms, users couldn't change the subcategory for a published term. This issue has been resolved, and users can now change the category and subcategory even when the term is Published/Draft state.

My Resources

In My Resources > Inbox > Alerts, users were not receiving alerts, there was no proper message displayed in the alerts even after selecting the 'send alerts on failure' option. This issue has been resolved, and users can now receive alerts with the proper message.

Advanced Tools 

• In the Advanced Tools > OvalEdge APIs> Schema API > Get connector schemas endpoint, when users were trying to fetch schemas of a particular connector Id, all the unwanted schemas were shown in the Response body. The issue has been resolved and only the active and inactive schemas are displayed in the Response body.
• In the Advanced Tools > OvalEdge API's > Connector API > Get all connectors endpoint, when users were trying to fetch all the connectors that are available in OvalEdge API, along with connectors, Jira and ServiceNow integrations were also fetched in the Response body. Now the issue has been resolved and only the connectors list is displayed in the Response body.
•  In the Advanced Tools > OvalEdge APIs> Business Glossary API, when users attempted to execute the APIs, no code was displayed. This issue has been displayed and users can now execute the API and get the code displayed as expected.
• In the Advanced Tools > OvalEdge APIs> Business Glossary API, Term Associated data was not displayed in the API response. This issue has been resolved, and the term-associated data is now displayed accurately in the API response.

Administration

• In the Administration > Connectors > Matillion Connector, users were unable to build lineage, view references in the Parent Job, access associations for Redshift queries, and encounter issues with source tables linked to queries. This issue has been resolved and users can now build lineage for the Matillion connector with Redshift as expected.
• In the Administration > Connector > Add New connector > DBT connector, when users attempted to run the process, it retrieved all runs executed in the DBT Cloud environment. This approach resulted in “OutOfMemoryException” errors due to the significant increase in heap memory usage. This issue has been resolved, and users can now exclude jobs using a comma separator in the edit connector setting option from 9 dots and they can crawl the remaining jobs successfully.
• In the Administration > Connectors > Add New Connector > Qlik Sense Connectors, there was no bridge option available while validating a connector. This issue has been resolved, and users can now see the bridge option before validating a Qlik Sense connector.
• In the Administration > Connectors > SAP Connector, users couldn't retrieve table names. When they attempted to search for tables using the table search functionality, the tables were not displayed or retrieved. This issue has been resolved and users can now retrieve table names as expected. 
• In the Administration > Connectors > Kafka, when users were trying to add a Kafka connector, there was no bridge option available on the validation page. The issue has been resolved, and users can now see the bridge option while validating a connector.
• In the Administration > Connectors > Power BI, there was a lack of lineage connection between Data Flows and Data sets. This issue has been resolved, and the connection has been established.
• In the Administration > Security > Snowflake Databases, users faced issues when applying masking policies while enabling column security and creating new policies. This issue has been resolved by allowing all cases and users can now apply the masking policy as expected. 
• In the Administration > System Settings, despite having set "oe.store.source.data" to "false," data continued to be displayed within the application. This issue has been resolved and users can no longer see the data.

Copyright © 2023, OvalEdge LLC, Peachtree Corners GA USA