Structured File Connector

Introduction

The StructuredFileConnector allows users to connect to XML and CSV files, read and write data, and transform the data as needed for various analytical, reporting, or data integration tasks.

OvalEdge uses the NFS path to place the files from the data source and crawl and profile.  

Connector Capabilities

Prerequisites

The following are the prerequisites to establish a connection between the connector and the OvalEdge application. 

1. Access Token Details
2. Service Account Permissions
3. Configure Environment Variables (Optional)

API Details

Read the XML file content using Java Libraries and prepare tables and columns.

Service Account with Minimum Permission

By default, the service account provided for the connector will be used for any user operations.

Establish Environment Variables (Optional)

This section describes the settings or instructions you should know before establishing a connection. If your environments have been configured, skip this step.

Configure Environment Names

The Environment Names allow you to select the environment configured for the specific connector from the dropdown list in the Add Connector pop-up window.
For consistency, you might want to consider crawling the same schema in both stage and production environments. The typical environments for crawling are PROD, STG, or Temporary, and may also include QA or other environments. Additionally, crawling a temporary environment can be useful for schema comparisons, which can later be deleted, especially during application upgrade assistance. 

Steps to configure the Environment on OvalEdge:

1. Navigate to Administration > System Settings
2. Select the Connector tab
3. Find the Key name “connector.environment”.
4. Enter the desired environment values (PROD, STG) in the value column. 
5. Click ✔ to save. 

Establish a connection

To establish the connection, complete the following steps:

1. Log into the OvalEdge application.
2. In the left menu, navigate to the Administration module and click on Connectors.
3. Click the + icon (New Connector), and then the Add Connector pop-up will appear.
4. Search/Click on the desired connector.
5. The specific connector details page is displayed.
   

   Fields	Details
   Connector Type	The selected connection type ‘StructuredFileConnector’ is displayed by default.  The drop-down list allows you to change the connector type if necessary. The fields associated with the selected connection type are displayed based on the selection.
   Credential Manager	Select the option from the drop-down list to indicate where you want to save your credentials: OE Credential Manager: When OvalEdge establishes a connection to the StructuredFileConnector database, the StructuredFileConnector connection is configured with the service account's basic username and password in real-time. Users must manually add the credentials if the OE Credential Manager option is selected. HashiCorp: The credentials are stored in the HashiCorp database server and fetched from HashiCorp to OvalEdge.   AWS Secrets Manager: The credentials are stored in the AWS Secrets Manager database server; OvalEdge fetches the credentials from the AWS Secrets Manager.  Azure Key Vault: Azure Key Vault allows for secure storage and strict access mechanisms of sensitive information such as tokens, passwords, certificates, API keys, and other confidential data.
   Credential Manager ConnId	Pick the specific ID you want in the Credential Manager ConnId field when you have more than one Credential Manager ID.
   License Add-Ons	All the connectors will have a Base Connector License by default, which allows you to crawl and profile to obtain metadata and statistical information from a data source.
   Connection Name	Enter a connection name for the Structured File Connector. You can specify a reference name to identify the Structured File connector in OvalEdge. Example: SFConnector
   FILE TYPE	Select the file type from the drop-down list. - .xml - .csv
   Connection File Path*	Specify the file path where XML or .csv are placed
   Meta Data Connection File Path*	Specify the file path where the files contain metadata of the above given path
   Select Bridge	The OvalEdge Bridge is necessary to enable OvalEdge to function as a SaaS application behind a customer's firewall.  When a bridge has been set up, it will be displayed in a dropdown menu. Users can select the required Bridge ID. The user can select "NO BRIDGE" when it is not configured. For more information, refer to Bridge Overview

6. Click on the Validate button to validate the connection details.
7. Click on the Save button to save the connection.  Alternatively, you can also directly click on the button that displays the Connection Settings pop-up window to configure the settings for the selected Connector. The Save & Configure button is displayed only for the Connectors for which the settings configuration is required. 

Connection Validation Errors

➤ If you have any issues creating a connection, please contact your assigned OvalEdge Customer Success Management (CSM) team.

Connector Settings 

Once the connection is successfully established, various settings are provided to fetch and analyze the information from the data source.  The connection settings include Crawler, Access Instruction, Business Glossary Settings, and Anomaly Detection Settings.

To view the Connector Settings page,

1. Go to the Connectors page.
2. From the 9- dots select the Settings option.
   
3. This will display the Connector Settings page where you can view all the connector setting options.
   
4. When you have finished making your desired changes, click on Save Changes. All setting changes will be applied to the metadata.

The following is a list of connection settings along with their corresponding descriptions:

For more information, refer to the Connector Settings. 

Catalog Files/Folders

The Crawl/Profile option allows you to crawl files from a data source and load them into the OvalEdge application. A new job gets initiated after selecting the connector and clicking Crawl/Profile. Once a job has succeeded, the top-level files and their columns are stored in Data Catalog > Files and Data Catalog > File Columns. The next level of data is stored in the File Manager.