Job workflow overview

The Job Workflow feature can be found in the Administration tool in the left navigation pane. The Job Workflow feature allows the user to organize, track, and create a series of recurring jobs. Notifications on the status of jobs can be set up and sent to the expert(s) identified. 

With workflows, a user can: 

• Run and troubleshoot jobs independently with real-time status feedback 
• Schedule workflows for jobs that should only run periodically
• Run multiple jobs in parallel for efficient testing 

The basic process for building a workflow job template is illustrated below:

Job Types

A Job type displays a set of actions that are currently available and can be a part of a job workflow.

• Crawler Connection: Establish a connection to your data source, and set up a  scheduler on the Connections to crawl all the schemas within the connection without any restriction. 
• Crawler: Setup the fetching of metadata from source systems at scheduled intervals. 
• Profile: Setup computing statistics and informative summaries about the data objects at scheduled intervals.
• SQL (Query): A User can schedule cataloged queries.
• Advanced Jobs: Schedule and execute advanced jobs such as building lineage, loading data dictionaries, and discovering relationships.
• Alerts: Data Alert is an automatic message sent in response to an event.
• Job workflow: Users can add existing Workflow templates as building blocks in another workflow. 

• Profile Unprofiled: Setup to compute statistics from the newly crawled or unprofiled tables.

• Data Quality Rule: Setup the data quality rules that are scheduled to run at regular intervals.

• Governance Policies: A user can include the Governance policies that are scheduled to be executed at given intervals.

• Process Unprocessed Queries: Select a Schema to process all the unprocessed queries(Failed Queries) that are used for building lineage.

Design Workflow

Multiple Jobs can be processed sequentially or in parallel by creating a workflow template. It is important to consider the following scenarios when building workflows

• Sequential Workflow: Progresses from one step to the next and does not step back.
• Parallel Workflow: Parallel Workflows allow some part of the workflow to run simultaneously.

Note:

1. If you remove a job or workflow template within the workflow space, the node(s) previously connected to those gets deleted and automatically connected to the previous node.
2. It is possible to launch several workflows and schedule to execute at a later time. Users can set notifications on workflows, to inform when a job completes.

Workflow space

A Workflow space provides a graphical way of linking together job templates to build a workflow template. 

Workflow Menu Bar

The Workflow Menu Bar is divided into three sections:

• File: It allows you to create, save, and close a workflow.
• Job: It allows you to review schedules set up for the selected template. Also, it allows users to run a saved job and display the history of jobs that have run in the past for the selected workflow.
• Setting: It allows you to review any notification operations users have set up. Also, you can select and Update Experts to send notifications as a system alert.

Note: All the Workflow operations are also available as easy to use visual shortcuts below the ribbon. 

Create a Workflow

To create a new workflow job template:

1. Click the Job Workflow tab from the object browser and select Add New Job on the Job Workflow information page to create a new workflow job template. 
2. Click the Job types and choose the corresponding jobs to be added to the workflow. 
3. Drag, and drop the Jobs in the workflow space and enter a Job flow name.
4. Save the workflow and close the Workflow space to return to the Job information page of the newly saved template.

Schedules

Scheduling a workflow allows the user to run a saved workflow at a user-defined time. 

Add a schedule to the existing workflow:

1. Click the Job workflow tab from the object browser and select a saved Job Flow name from the Job workflow information page. Hover over the Job menu bar and select Schedule Job from the drop-down list.
2. Choose a specific hour/ day/ week/ month/ year from the dropdown list to define the schedule and click schedule and the defined schedule is saved to the workflow.

See the list of all the scheduled tasks displayed in the Schedule module. For more information on Schedules refer to the Schedule article.

History of Jobs

The history of jobs allows users to see the tasks that have been successfully completed in a workflow.

1. Click the Job workflow tab from the object browser and select a saved Job Flow Name from the Job workflow information page. Hover over the Job menu bar and select History from the drop-down list.
2. Alternatively, on the Job workflow information page, Click on the clock icon, and it will navigate to Jobs History for chosen workflow.

Work with Notifications

Clicking on Notifications allows the user to review any notification integrations they have set up. When a job Succeeds/ Fails/ Partially Succeeds, a notification is sent using the configuration defined by the Notifier. Notifications can send as a system alert or an email. 

To set a notification:

1. Click the Job workflow tab from the object browser and select a saved Job Flow Name from the Job workflow information page. Hover over the Job menu bar and select Settings from the drop-down list.
2. Select Experts and choose the name of the expert to be notified.
3. Click update experts to save the name of the assigned expert to this workflow.
   
4. Select Notifications from settings and choose a type of notification from the choice. When to send notification on error or on success or on partial success.
5. Click Notify Experts to save and add the notification.

Execute the workflow

To execute/ run the workflow immediately,

1. Click the Job workflow tab from the object browser and select a saved Job Flow Name from the Job workflow information page.
2. Click 9 dots to Execute Now to run the workflow.
   Alternatively, click a saved workflow name and hover over the Job menu bar, and select Run Job. 

View Completed Jobs

To view a list of all workflow jobs that have been scheduled and executed, select the Jobs tab from the object browser to get various details about the job itself.

1. Select the Jobs tab from the object browser and click Job ID, a unique number used to identify each job initiated and executed. Click on it to get a graphical representation of the job lineage as shown below.

A graph is available only for the jobs triggered through the Job workflow. 

• Click on the Logs to know the execution steps history. We have the filter on the Log Value window in the form of INFO, WARNING, and ERROR. This will help users to find specific information when their job ends.

Workflow Status

The job workflow can have one of the following job step states

• INIT - The Job has been initiated.
• RUNNING - Jobs are being processed in the submitted order 
• WAITING - Submitted Job is in Queue to get executed 
• SUCCESS - An invoked task is completed successfully 
• ERROR - The job has failed. Check the error message in the Logs 
• HOLD - The initiated job has been kept on hold. 
• KILLED - The initiated job has been killed.
• PARTIAL SUCCESS - The initiated job has been completed partially.

Role-Based Access Controls

To create, edit, and delete a saved workflow job, you must be granted permission to all the resources used in a workflow. You must also have access to execute a job template to add it to a workflow job template.

For more information on role-based permissions, refer to Manage User and Roles article.