OvalEdge Migration Process
      Back to home
      1. Knowledge Base
      2. Architecture, Security & Releases
      3. OvalEdge Migration Process

      Release6.3.4 Migration Process for Container-Based Model

      This document guides migrating from previous versions to the latest Release6.3.4 version.

      Prerequisites for Migration

      Before initiating the migration process, ensure that the following prerequisites are met:

      Supporting Application Configurations 

      • Upgrade Infrastructure (Optional)

        Ensure that the applications and the operating system support the upgraded amount of RAM and have sufficient space on the new SSD for optimal performance.

      • Upgrade Database Version

        Upgrade the database to the below-mentioned versions for software performance and security. The previous versions of MySQL are no longer supported. 

      • Upgrade MySQL Version to 8.0.4x or higher. For more information, refer to
        Upgrading AWS MySQL 5.7 to 8.0.x
        • Note: The same process applies when upgrading MySQL from 8.0.3x to 8.0.4x.
      • Upgrade MariaDB Version to 10.6.x or 10.11.x. 

      Pre Upgradation 

      Take Database Backup

      Ensure a comprehensive backup of the Database (MySQL or MariaDB) is taken. 

      For more information, OvalEdgeDB Backup Process

      Table Row Count Validation

      Validate the row count of all database entities to confirm the success of the migration process. 

      Detailed instructions are available in the DB Migration Checklist

      Upgrade or Install ElasticSearch 


      Upgrade ElasticSearch to 7.17.x for improved search capabilities within the OvalEdge. Refer to the document for more information.

      Installing Elasticsearch on Container Systems

      Application Backups

      Before commencing the migration, execute the following backup and validation procedures:

      Database Backup

      Before starting the migration, ensure a comprehensive backup of the Database (MySQL or MariaDB) is taken.

      Image Backup 

      Backing up the image ensures that if the primary image becomes corrupted or lost during migration, it can be rolled back to the previous state if needed.

      Helm Charts Backup

      Helm charts contain the configuration details necessary to deploy and manage applications within Kubernetes clusters. Backing up these charts ensures the deployment environment is protected in case of any issues during migration.

      System Settings Backup

      Take the backup of the system settings table from the OvalEdge database in XLSX format to ensure the configured default settings are available to reflect in the new version.

      For detailed instructions on taking table backups to CSV/XLSX, refer to the MySQL Table Backup to CSV/XLSX Guide

      Migration Process

      For Container-based systems, the migration process involves updating the YAML (value.yaml) properties by replacing the older image name with a new one and uploading lineage and CSP jars if they are read from EFS. Refer to the document for more information EFS Volume Mount with Access point.


      Note: Lineage and csp-lib jars should be placed in third-party jars mounted on the EFS service, as referred to in the EFS Mount document.

      Governance App Configurations 

      These configurations are crucial to redirect users to governance-related applications using email or direct links.  Refer to the document to configure the Governance application. Governance App Configuration

      Configure One Pod at a Time

      It's crucial to emphasize bringing one pod at a time.

      As shown in the screenshot below, the job pod should be set to zero during deployment.

      The jobdeployment.yaml & uideployment.yaml file should contain the below parameters as comments, as shown in the screenshot, until the migration process is complete. 


      Note: Post Migration, uncomment the above parameters and restart the deployment. 

      Post Migration Checks

      Tables & Row Count Validation

      Conduct a thorough verification of Tables & data row counts in the post-migration environment.

      Ensure that the actual counts match the pre-upgradation counts to identify and rectify any discrepancies. This will serve as a key indicator of a successful migration. 

      The SQL scripts can be used to validate row and table counts.

      Download the scripts from the following S3 link: Tables and Row Count Validation Scripts.

      Verify Application

      Verify that the latest enhancements and features are successfully deployed in the application post-migration.

      Run the Advance Jobs

      This involves running the Advance Jobs, which are essential processes for ensuring the proper functioning of OvalEdge in its upgraded state. 

      For more information, please refer to the Appendix A below.

      Verify System Settings

      The migration process does not mandate adjustments to any system settings. However, it's important to note that for Release6.3.4, new system settings have been introduced, some have been deprecated, and others have undergone modifications. For more information, please refer to the Appendix B below.

      Appendix A: Advanced Jobs

      After migrating to Release6.3.4, the following advanced jobs must be run

      Run the same advanced job that was run for Release6.3 migration and additionally run the below listed advanced jobs associated with Release6.3.4. 

      A1: 6.3.4 Advanced Job Checklist 

      Advanced Job Name

      Description

      Migration_JobtoInactivate_Datasets

      This advanced job is used to make the child dataset(s) inactive when their parent dataset is inactive.
      Previously, the child dataset(s) remained active and visible in the Data Catalog under code associations, even when their parent dataset was inactive. However, the child datasets will now become inactive after running this job.
      Note:
      When to run this job:
      This migration job has to run once if users migrate from any release version to R6.3.4.

      DQTAT Execution Overview

      This job analyzes the Data Quality Test Suite, highlighting the status of each Data Quality Rule (DQR) executed in the current release. The job generates a summary of passed and failed rules, listing associated objects for each failed rule. This overview enables swift identification of potential data quality issues and assists teams in pinpointing areas that may require additional focus or remediation.

      Fetching Labels for Synapse Columns

      This job helps fetch the Synapse labels at table level and column level.

      Advance Job for Inserting data into the Lineage table

      This job allows the insertion of data into the Lineage Table to build a Connection Dashboard view.

      FME TO CARTO VISTA Lineage

      This job will establish lineage from FME to Carto Vista based on the FME Components.

      Report Crawl Analysis

      This job facilitates the analysis of activated and inactivated objects during the second crawl of the Power BI server. Upon running the job, a report is generated that provides details about the updated activated and inactivated objects of the Power BI server in OvalEdge, enabling users to track and manage object status effectively.

      Appendix B: New System Settings Added in Release6.3.4

      The latest release introduces new configurations that provide users with even greater control over the application's behavior. The newly added configurations are shown below.

      Key

      Description

      Impacted Modules in OvalEdge

      dataquality.associatedobjects.tables.limit

      Set the maximum number of objects (tables and table columns) that can be associated with a Data Quality Rule.

      Parameters:

      The default value is set to 1000.

      Enter the value in the field provided.

      Minimum entered value is 1.

      Maximum entered value is 1000.

      Data Quality

      dataquality.associatedobjects.files.limit

      Set the maximum number of objects (files and file columns) that can be associated with a Data Quality Rule.

      Parameters:

      The default value is set to 20.

      Enter the value in the field provided.

      Minimum entered value is 1.

      Maximum entered value is 250.

      Data Quality

      oe.profile.importancescore.threshold

      Configure to define a threshold value for the importance score of data assets in the profiling process. Only assets with an importance score equal to or greater than this threshold will be included in the profiling. 


      Parameters:

      The default value is 0.

      Enter the value in the field provided.

      The value should be in the range of 0-100.

      In Data Catalog > Tables, Table Columns.

      max.character.count.for.text.custom.fields

      Specify the maximum character count allowed for text custom fields.

      Parameters:

      The default value is 1000 characters.

      The accepted value is < 5000.

      Custom fields are available in the Data Catalog, Business Glossary, Tags, Data Quality Recommendations, ROPA Report, and ROPA Processing Activities.

      ai.description.enable

      Configure to show/hide the 'Generate with AI' icon for the Business, Technical, and Detail descriptions.

      Parameters:

      The default value is True.

      If set to True, show the 'Generate with AI' icon for the Business, Technical, and Detail descriptions.

      If set to False, hide the 'Generate with AI' icon for the Business, Technical, and Detail descriptions.
      • Business and Technical Descriptions in Data Catalog.
      • Business and Detail Descriptions in Business Glossary.

      home.lastlogin.enable

      Configure to show/hide the user's last successful login date and the number of unsuccessful login attempts between the current and the previous successful login.

      Parameters:

      The default value is 'False'.

      If set to 'True,' show the last successful login date and the number of unsuccessful login attempts between the current and the previous successful login.

      If set to 'False,' hide the last successful login date and the number of unsuccessful login attempts between the current and the previous successful login.

      Login page

      login.warning.enable

      Configure to show/hide the banner. Upon clicking Okay, users can enter their User ID and Password to proceed with the Login.

      Parameters:

      The default value is 'False'.

      If set to True, show the banner.

      If set to False, hide the banner.

      Login page

      login.warning.description

      The specified description in the Value column shows on the banner if login.warning.enable is set to 'true.'

      The maximum number of characters supported is 2000.

      Parameters:

      The default value is 'Empty'.

      If the Value column contains the content, it shows on the banner.

      If the Value column is 'Empty,' the banner is not shown.

      Login page

      Copyright © 2025, OvalEdge LLC, Peachtree Corners, GA USA.