Elasticsearch Connector

It provides support for crawling database objects and profiling of sample data.

11-4

The connectivity to the Elasticsearch connector is via the Elasticsearch Rest Client. The connector currently supports the following versions of Elasticsearch:

The drivers used by the connector are given below:

Driver / API

Version

Details

Elasticsearch Rest Client

7.16.2and above

https://mvnrepository.com/artifact/org.elasticsearch.client/elasticsearch-rest-client/7.16.2

Note: Latest version 8.3.1

Pre-requisites

To use the Elasticsearch Connector, the details specified in the following section should be available.

  • An admin/service account for crawling and profiling. 
  • The minimum privileges required are:

Operation 

Access Permission

Connection validate

READ

Crawl schemas

READ

Crawl tables

READ

Profile schemas, tables

READ

Technical Specifications

The connector capabilities are shown below:

Crawling

Feature

Supported Objects

Remarks


Crawling

Tables

 

Table columns

All data types 

Views

 

Profiling

Feature

Support

Remarks

Table Profiling

Row count, Columns count, View sample data

Supports all data types

View Profiling

Row count, Columns count, View sample data

View is treated as a table for profiling purposes

Column Profiling

Min, Max, Null count, distinct, top 50 values

 

Sample Profiling 

Supported

 

Connection Details

The following connection setting details should be added for connecting to an Elasticsearch database:

ES manage connection

 

Property

Details

Database Type

Elastic Search

License Type

Standard

Connection Name

Select a Connection name for the Elasticsearch  database. The name that you specify is a reference name to easily identify the Elasticsearch database connection in OvalEdge.

 Example: Elastic Search Connection DB1

Host/ IP /AES Endpoint

Database instance URL
Example: https://search-esmckinsey-iuko6rg7qqyrc7ovf6au5bskxe.us-east-1.es.amazonaws.com/

Username 

User account login credential

Password 

Password for the user

Role-based Authentication 

Role based ES

The following are the field attributes required for the connection of Elasticsearch for role-based authentication.

Property

Details

Region

Region of database

Is Role based Auth(Y/N)

provide role ARN

Role ARN

Region of database

Access key

Access Key

Secret key 

Secret Key

Once connectivity is established, additional configurations for crawling and profiling can be specified:

Crawler configurations Settings

Details

Tables, views and Columns

If the checkbox is selected it will crawl the tables and columns in Elasticsearch.

Include Table

Regex

It catalogs the table based on the regex search pattern added.

Exclude Table Regex

It doesn’t catalog the table based on the regex search pattern added.

 

Profiler Settings:

Profiler Settings

Details

Profile Type

Sample - Sample profiling will be performed based on sample profile size Disabled - profiling is disabled completely. 

No. of threads

No of threads used for profiling

Query TimeOut

It is wait time for query response

Rowcount constraint

If it is checked the profiling performed based the rowcount limit

Profile rules

It will include/ exclude the tables to perform profiling based on regex search pattern added.