Knowledge Base

Connectors

RDBMS

AWS Aurora

AWS Aurora is a fully managed relational database engine. Aurora is compatible with MySQL and PostgreSQL, allowing users to leverage existing tools and applications.

JDBC driver connects to the data source to crawl, profile, and build lineage.

unnamed (6)

This connector document should be used along with the Generic Features of Connectors document which covers the generic features and settings (Establishing Connection, Connector Parameters, Connector Settings, etc.) of the OvalEdge connectors that are common. This document outlines the specific connector information only.

Connector Characteristics

Connector Category	RDBMS
Connectivity	JDBC
Connector Version [OvalEdge Connector Version Information]	7.0
AWS Aurora Versions Supported [Data Source Version from which OvalEdge started providing support]	5.7.12 onwards
OvalEdge Releases Supported (Available from) [OvalEdge Release Version in which the connector was included in the OvalEdge Application]	7.0 onwards

Supported Features

Crawling of Metadata Objects

Schemas, Tables, Table Columns, Views

Stored Procedures, Functions, Triggers and Relationships

Crawling of Query logs

Metadata Source

Information Schema and My SQL Commands.

Profiling

Table Statistics: Row Count, Column Count, Density, Null Density

Column Statistics: Min, Max, Null Values, Distinct Values, Top Values

Types of profiling: Sample Profiling

Full Profiling

Auto Profiling

Crawling of Query Logs

Supported

Data Preview

Supported

Data Lineage

Auto Lineage via parsing Stored Procedures, Functions & Views		Supported
Auto Lineage via parsing Query Logs		Supported
Lineage Levels	Table Lineage	Supported
	Column Lineage	Supported

Data Quality Module

Not Supported

DAM (Data Access Management) Support

Not Supported

Bridge Support

Supported

Query Sheet Support

(Run simple queries)

Supported

Crawl of Usage Statistics (Source System)

Not Supported

Certifications at Source (Source System)

Not Supported

Prerequisites (Prepare AWS Aurora Environment)

The following are the prerequisites required for establishing a connection:

Driver Details

JDBC driver is provided by default over the OvalEdge installation. In case it needs to be changed, add My SQL and Postgres SQL drivers into the OvalEdge Jar path (/home/ovaledge/jarpath) to communicate with the AWS Aurora database.

Driver

Version

Details

My SQL: JDBC Driver

8.2.0

https://mvnrepository.com/artifact/mysql/mysql-connector-java/5.1.29

Postgres SQL: JDBC Driver

42.2.14

https://mvnrepository.com/artifact/org.postgresql/postgresql/

Note: The latest version is 42.2.23

Service Account with Minimum Permissions

Create an account to connect to OvalEdge with the following permissions to crawl metadata into OvalEdge. Required permissions and corresponding My SQL and PostgreSQL commands are given below for your reference:

Operation	Minimum Access Permission
Connection Validation	Read
Crawl Schemas, Tables	Read
Profile Schemas, Tables	Read

My SQL: The following commands are used to create an account to connect to OvalEdge:

Create a MySQL service account to establish a connection to OvalEdge.
Create a user named OE_User for localhost and set a password using the following command:

CREATE USER 'OE_User'@'localhost' IDENTIFIED BY 'password';

If you want to create a MySQL user and grant access from the remote machine with IP 192.168.10.100, run the following command:

CREATE USER 'OE_User'@'192.168.10.100' IDENTIFIED BY 'Password';

Grant Select access to the database connection validation and crawl schema and tables.

GRANT SELECT ON `Database_name`.* TO 'OE_User'@`localhost`;

The below query is to get the codes.

GRANT SHOW_ROUTINE ON *.* TO 'OE_USER'@'localhost';

PostgreSQL: The following commands are used to create an account to connect
to OvalEdge.

Create a PostgreSQL account to establish a connection to OvalEdge

CREATE USER OE_User WITH PASSWORD '<<password>>';

Grant Permission to get KEYS data

GRANT REFERENCES ON ALL TABLES IN SCHEMA ‘<<schemaname>>' TO ‘<<username>>' ;

Grant read access to database, schemas to preview data and query

GRANT CONNECT ON DATABASE "<<db-name>>" TO OE_User;

GRANT USAGE ON SCHEMA ‘<<schemaname>>'TO OE_role;

GRANT SELECT ON ALL TABLES IN SCHEMA "<<schemaname>>" TO OE_User;

GRANT SELECT ON ALL VIEWS IN SCHEMA "<<schemaname>>" TO OE_User;

GRANT SELECT ON ALL MATERIALIZED VIEWS IN SCHEMA "<<schemaname>>" TO OE_User;

Grant read access to future schemas

ALTER DEFAULT PRIVILEGES IN SCHEMA '<<schemaname>>' GRANT SELECT ON TABLES TO 'OE_USER’;

NOTE: The above query allows to read all tables created in the same DB in the future.

Configure Environment Variables (Optional)
This section describes the settings or instructions you should know before establishing a connection. If your environments have been configured, skip this step.
For more information, refer to the "Generic Features of Connectors" document.

Establish Connection

In the OvalEdge application, the AWS Aurora connector allows you to crawl the tables and table columns and build the lineage.

The "Generic Features of Connectors" document covers general settings and parameters common to most OvalEdge connectors. Here, we'll focus on the specific details required to configure the AWS Aurora connector.

AWS Aurora Specific Parameters

Field Name	Description
SSL Enabled*	Select SSL Enabled as true or false. True: SSL is activated, encrypting the data transferred between your computer and the server. False: SSL is not activated, so the data is not encrypted during transfer.
Use SSH Tunnel*	Select SSH Tunnel as true or false. True: An SSH tunnel is used, creating a secure, encrypted channel for the connection. False: An SSH tunnel is not used, so the connection does not have this additional layer of security.
AWS Aurora Repository type*	Select from the drop-down list. MySQL Postgres
Server*	Based on the selection, enter the name of the MySQL or Postgres database instance server URL accessible by the OvalEdge application. Format: <account>.mysqlcomputing.com
Port*	Enter the port number based on the AWS Aurora repository type. For MySQL, the port number is 3306 For Postgres, the port number is 5432
Database*	Please provide the name of the database associated with the connection. It could be either MySQL or Postgres.
Driver*	A JDBC driver is a Java library file with the extension .jar that connects to a database. Based on the selection, the driver details associated with the database will be auto-populated by default. Example: net.client.jdbc.SQLDriver org.postgresql.Driver
Username*	Based on the selected database, enter the Service Account to access the required database. *Note:* The application Username and Password will be displayed instead by default. However, the user must clear the details and enter the service account credentials for the repo type.
Password*	Enter the Service Account password to access the required database.

Connector Settings

The following are the AWS Aurora connector settings:

Crawler
Profiler
Query Policies
Access Instruction
Business Glossary Settings
Others

Limitations

S.No.

Description

Data Profiling for the following data type columns is not supported:

My SQL: “Long blob”, “blob”PostGreSQL : “bytea", "jsonb", "lseg", "polygon", "json", "box",

"macaddr", "line", "point", "array", "xid", "anyarray", "oidvector", "int2vector", "bytea", "tid", "pg_lsn","user-defined", "uuid", "hstore", "int4range", "int8range", "numrange", "tsrange", "tstzrange", "daterange”

Data Profiling limit for column length - supports up to 50,000.

Errors & Resolution

S.No.	Error Message(s)	Error Description / Resolution
1	Failed to establish a connection. Please check the credentials.	Error Description The primary reason for this error is usually incorrect or invalid credentials (username or password). Error Resolution Verify the username and password, check the connection string format, ensure network connectivity, review user access permissions, and validate the database endpoint.
2	Connection Timeout	Error Description The "Connection Timeout" error occurs when the connector is unable to establish a connection to the Aurora database within a specified time period. This can be due to network issues, incorrect configuration settings, or server unavailability. Error Resolution Check network connectivity, verify the endpoint and port, review firewall and security group settings, increase timeout settings, ensure the database server is running, reduce network latency, and consult logs for details.
3	Warning for unsupported data types while profiling	Error Description Warning for column SampleTable.column_binary but continuing operation. Skipping profile for unsupported datatype binary. Error Resolution Convert unsupported data types to supported ones if profiling is important. For example, change binary data to VARBINARY or BLOB if supported. Check for updates or plugins for your profiling tool that add support for more data types.

FAQs

Q1: How much does the driver cost?

A: The JDBC Driver for AWS Aurora is available at no additional charge.

Q2: Can I use the driver to access AWS Aurora from a Linux system?

A: Yes! You can use the driver to access AWS Aurora from Linux, Unix, and other non-Windows platforms.

Planned upgrades

S.No.	Expected Date	Upgrade Version	Description
NONE planned as of now