About OvalEdge 5.2
The OvalEdge 5.2 is the latest release improvised with the Lineage Graph to provide a clutter-free view and included the Remote Data Access Management functionality with the different managing permissions.
In this release, the existing query sheet process is enhanced where both the query generation and fetching of the results for that particular query are handled from the backend, and the results are directly displayed in the UI at Query Sheet Module. Along with the Query Sheet, the Service Desk is improvised with Service Request Templates to raise a request with the various 'Template Types' available for users to select based on Request Type and its object types.
This release also allows an admin user to create an Auxiliary domain and a new feature by the name Team Messages is included that allows users to first create a team of members and can add those teams as “Owner” or “Steward” for any object.
Along with the existing functionality enhancements, new integrations like Slack is included in the application and new connectors like DSEFS and MSAccess are included in the connector list to crawl the files and folders associated with the DSEFS Server/Machine and crawl the tables and columns associated with the “mdb” and “accdb” files located either in NFS, SFTP or FTP.
In this release, the critical and major bugs associated with the Global Search, Data Catalog, Business Glossary, Query Sheet, Governance Catalog, Dashboard, My Resources, Advanced Tools, Administration, and Connectors are fixed and working as expected.
Release Type Release Date
Major Release 15 November, 2021
2. What’s New
2.1. Business Glossary
2.1.1. New Relationship Type in Business Glossary
In the Business Glossary, earlier the users used to build the relationship between terms to terms only but now the existing functionality is enhanced to build the relationship with other objects like (table, table column, file, file column, chart, chart column, query) with the following additional relationship types to analyze the data.
- contains → is contained in
- related to → contrasts with
- is synonym to → is synonym to
- calculates → is calculated from
- replaced by → replaced from
- filtered by → filters
- Is before → Is after
- Duplicates
2.1.2. Algorithm to run AI for all the Terms in Domain
The new algorithm is included in the OvalEdge to execute all the terms defined and belonging to one particular domain. When the user clicks on the option of ‘Run Recommendation on unclassified schema”, it will run the jobs and trigger the algorithm to run all the terms by identifying that domain ID.
2.1.3. RACI Matrix
In the OvalEdge, the RACI tab is used to assign one or multiple user roles to analyze, validate, approve or reject the term. Once the user roles are assigned, then that particular user must select the ‘Initiate Workflow’ button to complete the approval process and publish the term. Here the Business Glossary- Term is also enhanced with a new RACI feature to add multiple users under different predefined (R- Responsible, A- Accountable, C- Consulted, I-Informed) Roles to publish the term with the RACI assigned users.
2.2. Dashboard
2.2.1. Data Lineage > Sankey
View Data Lineage Graph: In the OvalEdge the existing Sankey Lineage Graph is enhanced to provide a clutter-free view. Apart from the existing viewing capabilities, viewing Data Lineage graphs has become more user-friendly.
The following are the features included in this enhancement:
- Zoom in Zoom out: For more readability and visualizing, users can now zoom in and zoom out in the lineage graph and are capable of having the closure and clear visuals of the lineage graph.
- Dragging: This feature is enabled in place of the horizontal and vertical scroll bars. Here the users can easily drag into the lineage dashboard as per the needs.
- 3 Dots: The contextual menu associated with each object is provided with many more options to perform in the data lineage graph. Here the user can click on 3 dots to,
- Copy to clipboard: It is to copy the object name with connection and schema name
- View Association: It is used to view the dataset of a query, however, it is disabled for objects.
- Correct the lineage: It is used to show If any lineage is not correctly built
- Report Missing Lineage: It is used to notify missing lineage using service request missing object using object id.
2.2.2. Data Lineage > Flowchart
In the OvalEdge the existing Data Lineage is enhanced to view the data in the form of flow chart representation along with the Sankey diagram. Here the users can click on the flowchart tab to view the data lineage in the flowchart diagram and can click on the Sankey diagram tab to view the data lineage in the Sankey format.
Screenshot: Sample Data Lineage Flow Chart
Display of Primary and Foreign Key Columns: Currently, the columns associated with the Primary Key/Foreign Key are not being displayed in the UI, it was getting difficult for users to identify whether the particular column was the Primary Key/Foreign Key. Now, this functionality is improved by adding a new label, “Key” in the Column Details which displays whether the column is a Primary Key/Foreign Key and if the column is neither the Primary Key nor Foreign Key it just displays as blank.
2.4. Administration
2.4.1. Crawler > Remote Data Access Management
Remote Data Access Management (RDAM) is the concept of connecting to the remote connection and managing the roles, users, and permission of remote schema connection through OvalEdge. For example data, permission is maintained in the snowflake, hadoop for tables or reports. It's quite devious work for database administrators to assign these permissions. Also, there is lots of PII information that needs to be protected for certain groups, to ease this process RDAM is introduced.
Remote Data Access Management has three ways for connecting a remote database
- None: When a user crawls any schema, all the users and roles from the remote source will come into the Remote Users tab, and the Remote Roles tab in the Administration > Users & Roles.
- Remote System is a master: In the Remote Access tab, the user selects an option of a Remote system is the master, and when a user crawls a remote connection, all the users and roles available in remote source pertaining to that schema connection are displayed in the OvalEdge (Administration - > Users & Roles screen). See, User Manual 5.2 Users & Roles.
- At the time of crawling the user permission available on that schema will also be reflected in the Users & Roles | Remote users and Remote roles tab. Users will be able to log in with that user's default password, then the user can change it on the first login.
- When this option is selected the admin users cannot create, update or delete the users or roles will also be reflected in the Security, Schema, and Tables tab.
- OvalEdge is a master: When OvalEdge is the master, users can assign Roles and User based permissions to Objects. For that admin, users can use the existing Users and Roles or it can create new Users and Roles and then assign.
- At the time of Crawling, users and roles assigned to the schema are displayed.
- When this option is selected the admin users can create, update or delete the users or roles. This will get reflected or added in remote sources as well. It also considers the roles permissions and schema permissions. Security schema level permission can be updated from OvalEdge
For more details, please refer to the User Manual 5.2 Crawler > Remote Data Access Management.
2.4.2. Security > Domains
Auxiliary Domain: In the Administration > Security > Domains > Auxiliary Domain, while creating a new domain, an admin user is provided with the option to create an Auxiliary domain. Here the primary domain will have primary terms and similarly, an auxiliary domain will have one or many auxiliary terms created. At the time of associating the terms to any of the objects(Schema, Database, Table, Table Column, File, File Columns), the primary term can be mapped to the objects that belong to the primary domain and whereas one or more auxiliary terms can be assigned to the auxiliary domain.
2.4.3. Users & Roles > Team
In the Users & Roles > Team, a new feature by the name Team Messages is included that allows users to first create a team of members and can add those teams as “Owner” or “Steward” for any object. Here in the Team messages, users are invited into one or more teams and can collaborate with individuals or teams.
2.4.4. Audit Trails
Data Retention: The new advanced job is created to dispose of the audit logs for better performance of the application. The Audit Archive feature includes the Data Retention period to archive the logs for over a period of time, it could be a month or a year. Once the retention period for a particular data set expires, it can be deleted or moved and the users have the option to archive the logs in the CSV or SQL format. Here the Audit Archive page in configuration displays the current row count, days to retention, archive before delete, CSV, SQL, and last archive date.
2.5. Advanced Tools
2.5.1. Build Auto Lineage
In the Advanced Tools > Build Auto Lineage, the Lineage Refactoring extends the communication of OvalEdge Lab and Client Environment: The new feature export of lineages in the form of ZIP or JSON to the OvalEdge Labs is implemented to investigate the failed lineage in the clients’ environment. Here instead of accessing or requesting access for the client machine, we can now just export the failed lineage and investigate in the OvalEdge Lab. After the investigation, the corrected lineage and/or manually built lineage are imported into the client’s environment. Along with the export functionality, the existing matching algorithm, for the tables, has been improvised to build the lineage. The new option “Build Lineage for All Source Code” is included on the Build Lineage page to build the lineage for all the source codes existing or for that connection. The new column “Context” has been included in the Temp Table Correction page to merge the temp tables more efficiently.
2.5.2. Compare Files
In the Advanced Tools > Compare Files, the new functionality for comparing the CSV is been implemented to compare the two CSV (Comma Separated Value) files with the same header to identify the matching values and matching percentage associated with two files so that the user can easily analyze the missing values and quality of the data existing in the files.
2.6. Governance Catalog
In the Governance Catalog > Service Desk, the existing module is improvised with Service Request Templates to raise a request with the various 'Template Types' available for users to select based on Request Type and its object types. In the Service Desk page, users can click on the + icon and select the required service request template to raise a request.
For Example: If the Request type is 'Access Request for Table' then Templates can be ‘System Defined Template’ and ‘Customized Templates’:
- System Defined Templates: By default, there are 36 system-defined templates provided in the application to raise the request.
- Customized Template: Users can also raise a request by creating a custom template as per the requirement.
The Ovaledge Service Desk is also now capable of integrating with external applications such as the JIRA tool and ServiceNow workflow application. Here in the Manage Servicedeskpage, the 'Service Desk Integration' button is used to push or map the tickets from the OvalEdge application to the JIRA or Service now tool. The tickets which are mapped to external tools are displayed in the Service Desk page.
2.7. Configuration
2.7.1. Splunk Integration
In the Administration > Configuration > Splunk Integration, the new tab Splunk Integration is included to view the Splunk connection details. After configuring the Splunk connection parameters in Configuration > Splunk Integration tab, the OvalEdge application reads the Splunk configuration details from the configuration page in ExternalLogService and establishes the connection with Splunk. Once after establishing the connection with Splunk, the system posts the Audit logs messages to Splunk through Splunk Rest API. Here the users can edit/remove Splunk connection details through the Splunk Integration tab.
2.7.2. Slack Integration
In the Administration > Configuration > Slack Integration, The main notion of integrating OvalEdge with slack is the extensibility of alerts of jobs that are sent to clients through emails. Along with email notifications, if users want to get slack notifications of the OvalEdge jobs, they can get them through this integration. Direct messages can be sent to the desired slack channel from OvalEdge, but to achieve this a couple of slack parameters (optional and mandatory) need to be configured in OvalEdge.
2.8. Home Page
In the OvalEdge the existing Elastic Search functionality is enhanced with the configuration of keywords and user id. Here when the user searches for the data objects with the specific keyword and views the searched object the score is automatically recorded as 1 and based on the recorded score the sorting of the search results is performed and if the user configures the keyword manually then the system automatically increments the synonym score with the configured keyword and helps the users to view the search results based on the synonym score.
2.9. Query Sheet
Currently, the query generation process is handled from the frontend through the Query Sheet module and the results are fetched from the backend and used to display in the UI. Now, this process is enhanced where both the query generation and fetching of the results for that particular query are handled from the backend, and the results are directly displayed in the UI at Query Sheet Module.
- The new feature Query Policies is included in the Crawler specific connector settings, where the user can add permissions at the query level to generate the particular query type for the users enabled with the permissions and privileges.
- The new feature Remote Access (Data Access Authorization) is included in the crawler specific connector settings to ensure that the right user is accessing the query sheet. Here the system validates the provided user credentials and if the credentials are valid then that particular user will be accessing the query sheet.
- The new option Add Limit is included in the query to set the limit for the connectors having a huge row count which is beyond the regular limit and does not impact the system performance.
2.10. Connectors
2.10.1. Include Exclude Regex Schema
Currently, when the connection is crawled, the system stops the crawling when a schema has an issue on the remote side. In order to continue with the crawling process, the particular schema gets deleted, and then the crawling process starts over. To overcome this, the Crawler Page is included with the ‘Include Exclude Regex’ functionality at the schema level to skip schemas that incur an issue while in the crawling process.
2.10.2. DSEFS Connector
The new connector DSEFS (Datastax Enterprise File System) is included in the OvalEdge connector list to crawl the files and folders associated with the DSEFS Server/Machine. Here the folders and files existing in the DSEFS Server/Machine can be cataloged and supports profiling for various extension files like .CSV,.XLSX,.PARQUET, JSON,.TXT and etc.
2.10.3. MSAccess Connector
The new connector MSAccess(Microsoft Access) is included in the OvalEdge connector list to crawl the tables and columns associated with the “mdb '' and “accdb” files located either in NFS, SFTP or FTP. Here we will catalog all the tables, columns, queries in mdb and accdb files.
2.10.4. Power BI Connector on Premise Server
The existing connector Power BI is enhanced to support the client On Premise Server, where the clients using the On Premise Server can now crawl the reports & datasets and build the lineage to view the actual sources associated with the selected reports and datasets.
Copyright © 2019, OvalEdge LLC, Peachtree Corners GA USA