OvalEdge Migration Process
      Back to home
      1. Knowledge Base
      2. Architecture, Security & Releases
      3. OvalEdge Migration Process

      Release6.3.3 Migration Process for Container-Based Model

      This document guides migrating from previous versions to the latest Release6.3.3 version.

      Prerequisites for Migration

      Before initiating the migration process, ensure that the following prerequisites are met:

      Supporting Application Configurations 

      • Upgrade Infrastructure (Optional)

        Ensure that the applications and the operating system support the upgraded amount of RAM and have sufficient space on the new SSD for optimal performance.

      • Upgrade Database Version

        Upgrade the database to the below-mentioned versions for software performance and security. The previous versions of MySQL are no longer supported. 

      • Upgrade MariaDB Version to 10.6.x or 10.10.x. 

      Pre Upgradation 

      Take Database Backup

      Ensure a comprehensive backup of the Database (MySQL or MariaDB) is taken. 

      For more information, OvalEdgeDB Backup Process

      Table Row Count Validation

      Validate the row count of all database entities to confirm the success of the migration process. 

      Detailed instructions are available in the DB Migration Checklist

      Upgrade or Install ElasticSearch 


      Upgrade ElasticSearch to 7.17.x for improved search capabilities within the OvalEdge. Refer to the document for more information.

      Installing Elasticsearch on Container Systems

      Application Backups

      Before commencing the migration, execute the following backup and validation procedures:

      Database Backup

      Before starting the migration, ensure a comprehensive backup of the Database (MySQL or MariaDB) is taken.

      Image Backup 

      Backing up the image ensures that if the primary image becomes corrupted or lost during migration, it can be rolled back to the previous state if needed.

      Helm Charts Backup

      Helm charts contain the configuration details necessary to deploy and manage applications within Kubernetes clusters. Backing up these charts ensures the deployment environment is protected in case of any issues during migration.

      System Settings Backup

      Take the backup of the system settings table from the OvalEdge database in XLSX format to ensure the configured default settings are available to reflect in the new version.

      For detailed instructions on taking table backups to CSV/XLSX, refer to the MySQL Table Backup to CSV/XLSX Guide

      Migration Process

      For Container-based systems, the migration process involves updating the YAML (value.yaml) properties by replacing the older image name with a new one and uploading lineage and CSP jars if they are read from EFS. Refer to the document for more information EFS Volume Mount with Access point.


      Note: Lineage and csp-lib jars should be placed in third-party jars mounted on the EFS service, as referred to in the EFS Mount document.

      oasis.properties to Migrate from 6.3 to 6.3.3

      The oasis.properties file now includes a new configuration parameter, ldap.additional.userSearchBase=, specifically for organizations using LDAP authentication with multiple or additional user search bases. This parameter allows administrators to define an alternative search base for locating user information within the LDAP directory.

      For more information, refer to the 6.2 to 6.3.3 changes in oasis.properties file

      Governance App Configurations 

      These configurations are crucial to redirect users to governance-related applications using email or direct links.  Refer to the document to configure the Governance application. Governance App Configuration

      Configure One Pod at a Time

      It's crucial to emphasize bringing one pod at a time.

      As shown in the screenshot below, the job pod should be set to zero during deployment.

      The jobdeployment.yaml & uideployment.yaml file should contain the below parameters as comments, as shown in the screenshot, until the migration process is complete. 


      Note: Post Migration, uncomment the above parameters and restart the deployment. 

      Post Migration Checks

      Tables & Row Count Validation

      Conduct a thorough verification of Tables & data row counts in the post-migration environment.

      Ensure that the actual counts match the pre-upgradation counts to identify and rectify any discrepancies. This will serve as a key indicator of a successful migration. 

      The SQL scripts can be used to validate row and table counts.

      Download the scripts from the following S3 link: Tables and Row Count Validation Scripts.

      Verify Application

      Verify that the latest enhancements and features are successfully deployed in the application post-migration.

      Run the Advance Jobs

      This involves running the Advance Jobs, which are essential processes for ensuring the proper functioning of OvalEdge in its upgraded state. 

      For more information, please refer to the Appendix A below.

      Verify System Settings

      The migration process does not mandate adjustments to any system settings. However, it's important to note that for Release6.3.3, new system settings have been introduced, some have been deprecated, and others have undergone modifications. For more information, please refer to the Appendix A below.

      Appendix A: Advanced Jobs

      After migrating to Release6.3.3, the following advanced jobs must be run

      Run the same advanced job that was run for Release6.3 migration and additionally run the below listed advanced jobs associated with Release6.3.3. 

      A1: 6.3.3 Advanced Job Checklist 

      Advanced Job Name

      Description

      HDFS to HIVE Lineage

      This job establishes lineage from HDFS to Hive based on the location of Hive external tables, as specified in the custom fields of the Hive table.

      Update Ovalsight Details

      This advanced job is designed to update Ovalsight details and should be run for the first time following a migration.

      License Issue

      This advanced job, related to migration, aims to add additional attributes to certain existing connectors, fetch unique values for the connectors, and update the license accordingly.

      Appendix B: New System Settings Added in Release6.3.3

      The latest release introduces new configurations that provide users with even greater control over the application's behavior. The newly added configurations are shown below.

      Key

      Description

      Impacted Modules in OvalEdge

      ms.teams.integration.type

      Configure Microsoft Teams integration method for sending notifications.

      Parameters: The default value is Webhook.

      If you have updated Microsoft Teams to a version that no longer supports webhooks, switch the integration method to Workflows and set the value to Workflow.

      This setting impacts notifications.

      connections.order.destination.lineage

      Configure to define the connection order for destination tables based on which the schema, tables, or data objects are selected and displayed in priority order while executing a Snowflake to DocumentDB Lineage advance job.

      Parameters:

      The default value is empty.

      Enter the order of the connection ID separated by commas.

      It impacts the lineage between Snowflake and DocumentDB when built using an advanced job.

      hide.report.count.in.reportgroupfilter

      Configure to disable the display of the total Report count of a Report Group in the Report Group filter in the Data Catalog grid.

      Parameters:
      The default value is False.

      If set to True, the count will be shown.

      If set to False, the count will be hidden.

      This impacts the Report Group filter in the Data Catalog List Page for Reports.

      Copyright © 2024, OvalEdge LLC, Peachtree Corners, GA USA