It provides support for crawling database objects and profiling of sample data.
The connectivity to the Elasticsearch connector is via the Elasticsearch Rest Client. The connector currently supports the following versions of Elasticsearch:
The drivers used by the connector are given below:
|
Driver / API |
Version |
Details |
|
Elasticsearch Rest Client |
7.16.2and above |
https://mvnrepository.com/artifact/org.elasticsearch.client/elasticsearch-rest-client/7.16.2 Note: Latest version 8.3.1 |
Pre-requisites
To use the Elasticsearch Connector, the details specified in the following section should be available.
- An admin/service account for crawling and profiling.
- The minimum privileges required are:
|
Operation |
Access Permission |
|
Connection validate |
READ |
|
Crawl schemas |
READ |
|
Crawl tables |
READ |
|
Profile schemas, tables |
READ |
Technical Specifications
The connector capabilities are shown below:
Crawling
|
Feature |
Supported Objects |
Remarks |
Crawling |
Tables |
|
|
Table columns |
All data types |
|
|
Views |
Profiling
|
Feature |
Support |
Remarks |
|
Table Profiling |
Row count, Columns count, View sample data |
Supports all data types |
|
View Profiling |
Row count, Columns count, View sample data |
View is treated as a table for profiling purposes |
|
Column Profiling |
Min, Max, Null count, distinct, top 50 values |
|
|
Sample Profiling |
Supported |
Connection Details
The following connection setting details should be added for connecting to an Elasticsearch database:
|
Property |
Details |
|---|---|
|
Database Type |
Elastic Search |
|
License Type |
Standard |
|
Connection Name |
Select a Connection name for the Elasticsearch database. The name that you specify is a reference name to easily identify the Elasticsearch database connection in OvalEdge. Example: Elastic Search Connection DB1 |
|
Host/ IP /AES Endpoint |
Database instance URL |
|
Username |
User account login credential |
|
Password |
Password for the user |
Role-based Authentication
The following are the field attributes required for the connection of Elasticsearch for role-based authentication.
|
Property |
Details |
|
Region |
Region of database |
|
Is Role based Auth(Y/N) |
provide role ARN |
|
Role ARN |
Region of database |
|
Access key |
Access Key |
|
Secret key |
Secret Key |
Once connectivity is established, additional configurations for crawling and profiling can be specified:
|
Crawler configurations Settings |
Details |
|
Tables, views and Columns |
If the checkbox is selected it will crawl the tables and columns in Elasticsearch. |
|
Include Table Regex |
It catalogs the table based on the regex search pattern added. |
|
Exclude Table Regex |
It doesn’t catalog the table based on the regex search pattern added. |
Profiler Settings:
|
Profiler Settings |
Details |
|
Profile Type |
Sample - Sample profiling will be performed based on sample profile size Disabled - profiling is disabled completely. |
|
No. of threads |
No of threads used for profiling |
|
Query TimeOut |
It is wait time for query response |
|
Rowcount constraint |
If it is checked the profiling performed based the rowcount limit |
|
Profile rules |
It will include/ exclude the tables to perform profiling based on regex search pattern added. |