Overview of Data & Metadata Changes
The OvalEdge application consistently monitors and logs changes in both crawled metadata and profiled data after each crawling and profiling session. This is made feasible through a diverse set of Connectors that analyze the crawled and profiled statistics. These modifications are categorized into two main sections: Data Changes and Metadata Changes. Users can easily review and investigate these changes through the Data & Metadata Changes tool within the Advanced Tools.
Note: The connector can profile, only if the configuration ovaledge.handleprofile.stats of the System Settings is set to “true” by the OvalEgde Admin.
Data Changes
Within the Data Changes section, users gain a thorough understanding of all Tables and their respective Table Columns that have undergone any changes in the profiled data. This comprehensive overview empowers users to not only identify but also monitor specific changes in the profiled data statistics and summary values over time, contributing to the preservation of data integrity and consistency within the OvalEdge application.
Metadata Changes
In the Metadata Changes section, users access in-depth insights into any metadata changes pertaining to Tables & Table Columns, Codes, Reports, and APIs & API Attributes. This detailed information allows Author Users to actively monitor and comprehend the modifications of metadata within their data landscape.
By facilitating a comprehensive understanding of these changes, this section plays a pivotal role in enhancing data governance and management, ensuring a more effective and informed approach to handling metadata evolution within the OvalEdge application.
Navigating Data Changes
The Data Changes section highlights specific statistical modifications in a Table following the re-profiling of any Data Source. These alterations encompass changes in Row Count, Popularity Score, and Importance Score. Additionally, the section provides the latest comparison of Profiling Results for each table.
For example, if the Popularity Score of the TGT_employees table in Sqlserver shifts from 80 to 100 during re-profiling, the updated score of 100 will be presented in the Popularity Changes column.
Changes recorded in OvalEdge
The Unified View in the Data Changes section provides a comprehensive display of table modifications following the most recent profiling of any remote Data Source. This view presents details such as the Connector Name and Schema to which the Table belongs, along with the Table's Title and Business Description. Additionally, it includes the Last Profiled Date, indicating the specific date of the last profiling of the Data Source. All the fields displayed have at least one of the following features: filter, search, and sort.
The changes that are displayed for any Table are the following:
- Row Count Changes: The Row Count Changes indicate the latest changes in the number of rows in the table.
- Popularity Changes: The Popularity Score displays the number of times the users interacted with this data object by viewing, endorsing, commenting, adding tags, or querying the table. In the Unified View, it displays the latest change in Popularity Score after the Data Source was last profiled.
- Importance Changes: The Importance score shows how vital a data table is across the organization on a scale from 0-100 based on the number of relationships and the strength the data object has with other data objects. The importance score of a data object depends on the lineage and profiling actions. In the Unified View, it displays the latest change in Importance Score after the Data Source was last profiled.
Accessing Compare Profile Results
The Compare Profile Results feature of Advanced Tools enables users to observe alterations in a table's data across different profiling sessions. It also allows users to compare two data tables from distinct profiling periods, facilitating the reporting of changes in:
- Column Names
- Column Properties
- Column Attributes
- Column Statistics
Within Data Changes, selecting the table name redirects the user to the most recent profile comparison of the table. This comparison specifically entails the comparison of data changes between the last two profiling sessions conducted on the table.
The following data comparison is shown for any table:
- Profiled Date
- Row Count
- Popularity Score
- Importance Score
For the Table Columns corresponding to the table, we see the changes for:
|
Data |
Description |
|
Column Position |
The position change of any particular table column. |
|
Column Name |
The change in the column name since the last profiling session. |
|
Column Type |
The change in the column type, an example being the change from int to varchar. |
|
Top Values |
The changes in the Top 50 most occurring values of the column. |
|
Null Count |
The change in the number of nulls of any column. |
|
Distinct Count |
It refers to unique values that are not repeated in a column. The latest change in the value is shown here. |
|
Minimum Value |
It refers to the lowest value of all records in a column. The latest change in the value is shown here. |
|
Maximum Value |
It refers to the highest value of all records in a column. The latest change in the value is shown here. |
Navigating Metadata Changes
The Metadata Changes section within Data & Metadata Changes highlights changes to the metadata of various data objects. These changes become evident in the Metadata Changes section following the re-crawling of the data connection. The section specifically reveals modifications for four types of objects: Tables, Codes, Reports, and APIs. Users can conveniently navigate through the provided tabs to review the most recent changes subsequent to the last crawl.
Metadata Changes for Tables
The Metadata Changes-Tables section provides a comprehensive overview of all tables that have undergone metadata changes after the latest crawl. For every table in the Schemas of the crawled connectors, the unified view of Metadata Changes showcases the following details for the tables:
|
Metadata |
Description |
|
Type |
It displays the type of the table, which can be a Table, View or Materialized View. |
|
Connector Name |
It displays the Connection name having the table, which underwent metadata changes. |
|
Schema |
It displays the Schema name having the table, which underwent metadata changes. |
|
Table |
It displays the table name for which metadata changes can be seen. |
|
Title |
This column displays the latest title that is given to the table by any Meta-Write User. |
|
Business Description |
This showcases the latest Business Description edited for any table. |
|
Technical Description |
This showcases the latest Technical Description edited for any table. |
|
Last Crawled Date |
This displays the date on which the table was last crawled. |
|
Last Metasync Date |
This displays the date on which the changes in source metadata synced with the OvalEdge application through re-crawl. |
Metadata Changes for Table Columns
When a user selects a table from the Table Metadata Changes Unified View, they are redirected to another unified view which consists of all metadata changes of all the table columns corresponding to the table. Users can view the following latest details for every column after the latest crawl:
|
Metadata |
Description |
|
Column |
This column displays the latest column names of the table. |
|
Data Type |
The data type is essentially the type of data that the column holds. The view shows the latest change if the data type changes for that column. |
|
Column Length |
The Column Length column shows the change value in the length of the column in comparison to the previous value. It can be positive or negative (in case of a length decrease). |
|
Column Position |
The Column Position displays the latest position that has been assigned to the column after the re-crawl. |
|
Created On |
The Created on column shows the date when the table was first crawled. |
|
Technical Description |
The Technical Description column displays the latest changed Technical Description of that column. |
|
Remarks |
The Remarks column displays whether a column of the table has been added, modified or deleted. |
Metadata Changes for Codes
The Unified View of Metadata Changes for Codes displays all the metadata changes that crawled queries went through during the last crawl. It shows all the latest changes that the codes have gone through.
|
Fields |
Description |
|
Connector Name |
Displays the Connection name having the query in which the changes are made. |
|
Schema |
Displays the Schema Name having the queries, which underwent changes. |
|
Code Name |
Displays the Name of the Query. |
|
Title |
Displays the title of the Query. |
|
Code |
Displays the content of the Query. |
|
Business Description |
Displays the Business Description associated with the query. |
|
Technical Description |
Displays the Technical Description associated with the query. |
|
Last Crawled Date |
Displays the latest date when the database was crawled. |
|
Last Meta Sync Date |
Displays the date and time on which any data changes are made to the query on the source system. |
|
Version |
Displays the version of the query. |
Users also have the capability to search for queries within a specific timeframe by inputting the start and end dates through the Date Filter available in the Unified View.
Compare Codes
Upon selecting any query from the Unified View, a user is taken to another view which displays all the versions of that code along with certain details. Users can primarily see the date when the code version was last run, along with the Technical Description of the same time period.
The primary purpose of this view is to compare two versions of the same code using the ’Compare’ button provided on the screen. By clicking, it displays the complete code of both versions for the users to analyze.
Metadata Changes for Reports
The Unified View of Metadata Changes for Reports displays all the metadata changes that crawled reports went through after the last crawl. The details pertaining to each report that can be seen on the view are:
|
Fields |
Description |
|
Connector Name |
Displays the Connection name that the Reports are a part of. |
|
Report Group |
Displays the name of the Report Group, to which the Report belongs. |
|
Report Name |
Displays the name of the Report. |
|
Title |
Displays the title of the Report. |
|
Business Description |
Displays the Business Description associated with the Report. |
|
Technical Description |
Displays the Technical Description associated with the Report. |
|
Last Crawled Date |
Displays the latest date, on which the report was crawled. |
|
Last Meta Sync Date |
Displays the date and time on which any data changes are made to the report on the source system. |
|
Version |
Displays the version of the Report. |
Metadata Changes for APIs
The Unified View of Metadata Changes for APIs displays all the metadata changes that crawled APIs went through during the last crawl. The details pertaining to each API that can be seen on the view are:
|
Fields |
Description |
|
Connector Name |
Displays the Connection name that the APIs are a part of. |
|
API Group |
Displays the name of the API Group, to which the API belongs. |
|
API Name |
Displays the name of the API. |
|
Title |
Displays the title of the API. |
|
Business Description |
Displays the Business Description associated with the API. |
|
Technical Description |
Displays the Technical Description associated with the API. |
|
Last Crawled Date |
Displays the latest date, on which the API was crawled. |
|
Last Meta Sync Date |
Displays the date and time on which any data changes are made to the report on the source system. |
Search Functionality & Data Discovery
OvalEdge provides a powerful search functionality that enables effortless access to any information incorporated within the Unified View of Data & Metadata Changes.
Users can conveniently navigate the Unified page of the Data & Metadata Changes, utilizing a user-friendly List View format. The List View enhances the searching process by offering filters, search options, and sorting capabilities, contributing to a more comprehensive understanding of changes.
Filter: Users can narrow down the search results based on a group of selected attributes from the drop-down pre-defined filter attributes for the column and can utilize the search bar to find the desired column filter attributes.
Search: The search filter aids users in precisely pinpointing the data objects they want within the vast data ecosystem. Also, a conditional search button represented by the eight dots icon, is available next to the search field to further refine the search results by excluding/including keywords in search results.
Sort: Most of the fields in the OvalEdge application have the ‘Sort’ function allowing a user to arrange the data objects in ascending or descending order. Based on the first column sorting results, the contents of the rest of the columns can be further sorted.
Utilizing the provided options, users can effectively navigate and explore data and metadata changes of all data objects, enhancing their discovery process based on the types of data displayed within the columns.