Data Quality Rule Recommendations - A Deep Dive

Previously, users faced limitations when applying Data Quality Rule Recommendations. They could only target one type of object at a time, such as a Table, Table Column, File, or File Column, and only one specific object. Running recommendations on multiple objects simultaneously was not an option.

However, the Data Quality Rule Recommendation functionality has now been significantly improved. Users can build rule models that provide suggestions for multiple data objects, whether they are Tables or Files. This enhancement allows users to easily create, run, modify, and delete these models. Each model generates recommendations that users can either accept or ignore. Furthermore, users can measure the model's effectiveness using Model Metrics and apply recommendations based on compliance by choosing the appropriate Data Quality Template.

This article will delve into the details of these new capabilities, exploring how they enhance the user experience and improve data quality management.

Where do we use Rule Models? 

• Based on the Table or File: When users are unsure about what rule to apply to a Data Object, they can use rule models to get recommendations. This feature helps eliminate confusion and provides tailored suggestions for tables or files, ensuring that the right data quality rules are applied to the correct data objects.

• To Apply Compliance: When users need to apply compliance to a Table or File, rule models can be used to suggest rules that align with specific compliance attributes. For example, when running recommendations based on a Data Quality Template, such as HMDA, the system will recommend rules for columns that match attributes like Legal Entity Identifier. These recommendations are based on a matching score, helping users apply the necessary compliance measures efficiently.

How do we use the Rule Models?

 Users can manually run models to generate recommendations. They can edit Rule Models and modify them as needed. Within the Rule Models, users can view the recommendations for data objects and decide whether to apply or ignore them. Accepting the recommendations will convert them into Data Quality Rules.

How do the Rule Models Work?

The Rule Models are configured by users to generate recommendations for data objects. To begin, users select the type of recommendation they wish to run on the model. Based on this selection, the system generates appropriate recommendations. Each Rule Model is built by applying specific conditions, which guide the recommendation process.

Once configured, the model analyses the data objects according to the set conditions and generates a list of suggested rules. Users can then review these recommendations, choosing to accept or ignore each one. The Rule Models also provide metrics to measure their effectiveness, allowing users to adjust and refine the models for optimal performance. This streamlined process ensures that users can efficiently manage data quality across multiple objects simultaneously.

Security - Who can Create a Rule Model

The Rule Models are created by the author license user and role admin users will act like the admins.

Creating a New Rule Model

Two-Step Process

The creation of a new recommendation model is a two-step process. 

1. Model Configuration (Model Name, Model Description, and Recommendation Type)
2. Object Selection (Table or File)

Model Configuration  (Step 1)

In Step 1, the user will be configuring the Model Name, Model Description, and Recommendation Type. The model name should not include any special characters or spaces. All the fields are Mandatory.

Object Selection (Step 2)

Select Object Type: In this step, users need to select the Object Type before selecting the objects. It can be either a Table or a File. Based on the selected Object Type, objects will be populated.

Select Objects: In this step, users need to select the Objects based on the Object Type selection.

Note: The user will be able to select only the tables or the files, however, the suggestions will be based on both the table and table columns if the table is selected, and the file columns if the file is chosen.

Landing Page

Model Details 

The landing page displays a list of recommendation models. Running a model will give users recommendations for data objects. Users can accept/ignore rule recommendations on data objects. Each model will have model metrics where users will be able to see the performance of the model.

Rule Model Name

This column displays the Rule Model Name which is created by the user.

Description

This column displays the Rule Model Description which is given by the user.

Recommendation Type

This column displays the Recommendation Type which the user has selected. It can be AI, Data Quality Template, or AI & Data Quality Template based on the user selection

Status

This feature will show the status of each recommendation model

• New - When the Rule Model is created for the first time
• In Progress - When the Rule Model is identifying the recommendations
• Completed - When the Rule Model execution is completed successfully
• Failed - When the Rule Model execution is Failed

Last Run Date

This column will display the date and time of the last execution of the recommendation model.

Duration

This column shows the time duration for which the recommendation was running.

Created By 

This column tells the user who has created and configured the recommendation model.

Created On 

This column tells the user when the model was created. It tells the date and the time for better understanding. 

Updated By 

This column tells the user who has updated and configured the recommendation model. 

Updated On 

This column tells the user when the model was updated. It tells the date and the time for better understanding.

9 Dots Functionalities

• Run Rule Recommendation: This feature allows the user to Run the model and get rule recommendations for the data object. Running a model triggers a job in the backend.
• Delete Recommendation Model: This option allows the user to delete a particular recommendation model if not being in use.

Recommendation Model Summary Page

Recommendations for Data Objects

Whenever users click or enter the recommendation model, they will be shown the rule recommendations for the data objects selected.  

Object details along with the Recommended rules are shown to the user. Users can click the Thumbs up icon for the given recommendation, by doing this, the Recommendation will turn into a Data Quality Rule.

Other Landing page Functionalities 

• Bulk operations (Bulk Accept) within recommendation models: This feature enables users to perform bulk actions such as approval of recommendations. This feature will be handled through 9 dots, where the user selects Accept Recommendations.
• Rule Settings: This feature allows the users to set their preferences on the Rules like Sending Alerts on Failure, creating service Request on Failure, Avoid Reporting Duplicate Service Requests, Apply Caution to Downstream Objects Upon Failure, Add Failed Values to the Remediation Center, Avoid Duplicate Failures, Max Failed Values Limit, and Schedule.
• Model Metrics: Model metrics give an overview of the Rule Model to the user. Model metrics help the user to understand how a particular Rule Model is performing. It shows all the execution details and how many recommendations were accepted after each execution. Based on these details, the model creator can decide whether to modify the model or delete the Rule Model if not in use.

Integration with other Modules

Integration with Data Quality Rules

Terms with Recommendation: If any Rule Recommendations were accepted from the Rule Model, those rules which are created will be displayed here.

Integration with Notification

The executor will receive notifications in the following manner.