Azure Repos

Azure Repos is a part of Azure DevOps that offers version control tools for easy code management and change tracking.

OvalEdge uses Rest APIs to connect with Azure Repos as a data source. This helps crawl Files and Folders data objects.

Connector Capabilities

The connector capabilities are shown below:

Crawling

Prerequisites

The following are prerequisites for connecting to the Azure DevOps Repo:

Drivers/API

The APIs/drivers used by the connector are given below:

Configuring Environment Variables

Configuring environment names enables you to select the appropriate environment from the drop-down list when adding a connector. This allows for consistent crawling of schemas across different environments, such as production (PROD), staging (STG), or temporary environments. It also facilitates schema comparisons and assists in application upgrades by providing a temporary environment that can later be deleted if needed.

Before establishing a connection, it is important to configure the environment names for the specific connector. If your environments have already been configured, skip this step. 

Steps to Configure the Environment

1. Log in to the OvalEdge application.
2. Navigate to Administration >  System Settings.
3. Select the Connector tab.
4. Find the key name “connector.environment”.
   
5. Enter the desired environment values (PROD, STG) in the Value column.
6. Click ✔ to Save.

Service Account Permissions

An admin/service account is necessary for crawling and building lineage. The minimum privileges required are:

Establish a Connection

To connect to Azure DevOps Repo using the OvalEdge application, complete the following steps:

1. Log in to the OvalEdge application.
2. Navigate to Administration >  Connectors.
3. Click on the + (New Connector) icon.
   
4. The Add Connector pop-up window is displayed where you can search for the Azure DevOps Repo connector.
   
5. The Add Connector with connector type-specific details pop-up window is displayed. Enter the relevant information to configure the Azure DevOps Repo connection. Informational fields and the descriptions are listed below.
   Note: An asterisk (*) denotes a mandatory field required for establishing a connection.
   
   

   Field Name		Description
   Connector Type		This field allows you to select the connector from the drop-down list provided. By default, 'Azure DevOps Repo' is displayed as the selected connector type.
   Connector Settings
   Credential Manager*		Select the option from the drop-down list indicating where you want to save your credentials: OE Credential Manager: The connection is configured with the basic Username and Password of the service account in real-time when OvalEdge establishes a connection. Users need to add the credentials manually if the OE Credential Manager option is selected. HashiCorp: The credentials are stored in the HashiCorp database server and fetched from HashiCorp to OvalEdge. AWS Secrets Manager: The credentials are stored in the AWS Secrets Manager database server; OvalEdge fetches the credentials from the AWS Secrets Manager.  Azure Key Vault: Azure Key Vault allows for secure storage and strict access mechanisms of sensitive information such as tokens, passwords, certificates, API keys, and other confidential data.  For more information on Azure Key Vault, refer to Azure Key Vault. For more information on Credential Manager, refer to Credential Manager.
   License Add Ons		All the connectors will have a Base Connector License by default that allows you to crawl and profile to obtain the metadata and statistical information from a data source.  OvalEdge supports various License Add-Ons based on the connector’s functionality requirements. Data Quality: Select the Data Quality Add-On license to identify, report, and resolve the data quality issues for a connector whose data supports data quality, using DQ Rules/functions, Anomaly detection, Reports, and more.
   Credential Manager ConnId		When you have more than one Credential Manager ID, pick the specific ID you want in the Credential Manager ConnId field.
   Connector Environment		The Connector Environment drop-down list allows you to select the environment configured for the connector from the drop-down list.  For example, you can select PROD, or STG (based on the configured items in the OvalEdge configuration for the connector.environment). The purpose of the environment field is to help you identify which connector is connecting what type of system environment (Production, STG, or QA). Note: The steps to set up environment variables are explained in the Configuring Environment Variables section.
   Connector Name*		Provide a connector name for the Azure DevOps Repo database in OvalEdge. This name will serve as a reference for identifying the Azure DevOps database connection.  Example: "Azure DevOps_Connection_test"
   Organisation*		Specify the name of the organisation. An "organization" refers to the top-level container that contains projects for managing resources.
   Project*		Specify the name of the project. A "project" is a container for repositories. It serves as a way to organize and manage work within the Azure DevOps environment.
   Repository*		Specify the name of the Repository. A "repository" is a version control system (VCS) that serves as a central location for storing and managing source code, configuration files, and other development assets.
   Token*		Token generated from Azure DevOps Repo instance.
   Connection String		Set the Connection String toggle button to automatically get the details from the credentials provided. Alternatively, you can manually enter the string. Format: https://dev.azure.com/{Organization}/{Project}/_apis/git/repositories/{Repository} Example: https://dev.azure.com/aaa/xxx/_apis/git/repositories/xyz
   Default Governance Roles
   Steward*		Select the Steward from the drop-down list options.
   Custodian*		Select the Custodian from the drop-down list options.
   Owner*		Select the Owner from the drop-down list options.
   Governance Roles 4, 5, 6*		Select the respective user from the drop-down options. Note: The drop-down list displays all the configurable roles (single user or a team) as per the configurations made in the OvalEdge application: Administration > Security > Governance Roles section.
   Admin Roles
   Integration Admins*		To add Integration Admin Roles, search for or select one or more roles from the Integration Admin options, then click the Apply button. The responsibilities of the Integration Admin include configuring crawling and profiling settings for the connector, as well as deleting connectors, schemas, or data objects.
   Security and Governance Admins*		To add Security and Governance Admin roles, search for or select one or more roles from the list, and then click on the Apply button. The Security and Governance Admin is responsible for: Configuring role permissions for the connector and its associated data objects. Adding admins to set permissions for roles on the connector and its associated data objects. Updating governance roles. Creating custom fields. Developing Service Request templates for the connector. Creating Service Request approval workflows for the templates.
   No. of Archive Objects*		The number of archive objects indicates the number of recent metadata modifications made to a dataset at a remote/source location. By default, the archive objects feature is deactivated. However, users may enable it by clicking the Archive toggle button and specifying the number of objects they wish to archive.
   Select Bridge*		With the OvalEdge Bridge component, any cloud-hosted server can connect with any on-premise or public cloud data sources without modifying firewall rules. A bridge provides real-time control that makes it easy to manage data movement between any source and destination. For more information, refer to Bridge Overview

6. After entering all the required connection details explained above, select the appropriate option based on your preferences: 
   1. Validate: Click on the Validate button to verify the connection details. This ensures that the provided information is accurate and enables successful connection establishment.
   2. Save: Click on the Save button to store the connection details. Once saved, the connection will be added to the Connectors home page for easy access.
   3. Save & Configure: For certain Connectors that require additional configuration settings, click on the Save & Configure button. This will open the Connection Settings pop-up window, allowing you to configure the necessary settings before saving the connection.
7. Once the connection is validated and saved, it will be displayed on the Connectors home page.

Connection Validation Details

Connector Settings

Once the connection is successfully established, various settings are provided to fetch and analyze the information from the data source.  The connection settings include Crawler, Access Instruction, Business Glossary Settings, and Anomaly Detection Settings.

To view the Connector Settings page,

1. Go to the Connectors page.
2. From the 9- dots select the Settings option.
   
3. This will display the Connector Settings page, where you can view all the connector setting options.
   
4. When you have finished making your desired changes, click on Save Changes. All settings will be applied to the metadata.
   The following is a list of connection settings along with their corresponding descriptions:
   

   Connection Settings	Description
   Crawler	Crawler settings are configured to connect to a data source and collect and catalog all the data elements in the form of metadata.
   Access Instruction	Access Instruction allows the data owner to instruct others on using the objects in the application.
   Business Glossary Settings	The Business Glossary Settings provide flexibility and control over how users view and manage term association within the context of a business glossary at the connector level.
   Anomaly Detection Settings	Users can configure Anomaly Detection settings at the connector level. Custom settings for anomaly detection can be provided, or users can opt for the Global Settings configured from the system settings.

Catalog Files/Folders

The Crawl/Profile option allows you to crawl files from a data source and load them into the OvalEdge application. A new job gets initiated after selecting the connector and clicking Crawl/Profile. Once a job has succeeded, the top-level files and their columns are stored in Data Catalog > Files and Data Catalog > File Columns. The next level of data is stored in the File Manager.