Run Entity Consolidation

Canopy simplifies the process of consolidating entities by including a preloaded global ruleset or allowing the user to define their own consolidation rules.

βeta Feature
Run Consolidation dashboard and activity history is currently a beta feature.

Once queued, the progress bar will appear, providing the user the ability to supervise and stop the consolidation process, should they need.

The features of the dashboard include the following:

Stop Consolidation

An action for the user who wants to stop the consolidation.

Check Activity History

Click on the Check Activity History shortcut to filter only the activities associated with the current running job.

Job ID
The system creates a new Job ID each time a user runs consolidation. The user can filter and search the Activity History using this Job ID to see all activity associated with one run action.

Close Progress Bar

Clicking on the down arrow will close the progress bar.

View Progress

Users can view the steps and the progression of each step in the consolidation process. Consolidation status details will be displayed as each step is completed.

This process assumes that you have consolidation at least once and that you are not changing the consolidation rules.

1. Sign in to the project with a Project Administrator role.

2. Click Entities on the top navbar.

3. At the top right of the entities view, click on the Run Consolidation button at the top of the page and then click on Run in the modal to queue a background consolidation task.

   Running consolidation will always default to remembering previous master entities and manual merges.

4. Monitor the progress of consolidation displayed at the top of the page. See How to Monitor Consolidation.

5. Confirm consolidation has completed when the dashboard is no longer present and the status Entity consolidation completed appears at the bottom of the page.

This process assumes that a consolidation task is running. Users at the Review Manager level or higher have the flexibility to stop consolidation while it is running or being queued in order to make changes or perform other work.

1. Sign in to the project with a Project Administrator role.
2. Click Entities on the top navbar.
3. In the consolidation progress dashboard, click on the Stop Consolidation button on the left.
4. Confirm consolidation has stopped.

Consolidation will go through the following steps:

1. Queued -> Synchronize
2. Back Up
3. Delete - Present when user chooses to Reset Consolidation or Delete Manual Decisions.
4. Cluster
5. Check

Consolidation will start within 60 minutes after the user queues the background consolidation task. Every 20 minutes, a synchronization task is run and, in some cases, if it is still running, could delay consolidation from starting within the 60 minute interval. The duration of time the action remains in the queue is updated every minute.

Queued Dashboard Step

Once picked up for the queue, the first step in the consolidation process is to synchronize entities with Canopy’s consolidation database. If the consolidation database does not exist, it will be created. Otherwise, the system will incrementally complete the following actions:

1. make copies of new entities,
2. make adjustments to revised entities, and
3. delete deleted entities.

The count of updates (new + changed + deleted) will be shown in aggregate after they are synchronized.

Synchronizing is the working state for this step and shows the progress of the number of entities that are left to synchronize, counting down to zero(0).

Synchronizing Dashboard Step

At times, the system may pause the working state and the subtitle of the step will be prefixed with the text System Paused.

Synchronized is the finished state for this step and will display the final number of entities synchronized and the duration it took to perform the step.

Synchronized Dashboard Step

The Activity History will show more detailed information regarding how many entities of each type (new, changed, or deleted) were synchronized.

This step backs up the list of raw and consolidated entities in the event that Stop Consolidation is run, so that the system can revert to the state prior to running consolidation.

While backing up the entities, the subtitle of the step will indicate first that the raw entities are being backed up, and then the master (consolidated) entities.

Backing Up Dashboard Step

At times, the system may pause the working state and the subtitle of the step will be prefixed with the text System Paused.

Backed up is the finished state for this step and will display the count of entities backed up and the duration it took to complete the backup step.

Backed Up Dashboard Step

If the user selects Delete Manual Decisions and/or Reset Consolidation in their consolidation rules, then this step will be visible and show the status of deleting the current consolidation results.

• Delete consolidated list only
• Delete consolidated list and decisions
• Delete manual decisions only

Consolidation Rules

Deleting Dashboard Step

At times, the system may pause the working state and the subtitle of the step will be prefixed with the text System Paused.

Clustering is the first step implementing the consolidation process. Clustering consists of two parts:

1. First, this process deduplicates the entities by clustering entities that have the same hash value.
2. Following deduplication, the entities are then clustered per the consolidation rules.

Canopy can estimate the maximum number of clusters and provide an indication of the consolidation progress.

When consolidation is complete, the actual number of clusters will be noted. After creating this checkpoint, the clustering step begins. Entities will be deduplicated and clustered according to Canopy’s default global ruleset. The maximum number of clusters can be estimated, so an indication of the consolidation progress will be provided. When consolidation is complete, the actual number of clusters will be noted.

At times, the system may pause the working state and the subtitle of the step will be prefixed with the text System Paused.

In the final step, the conflict checking process attempts to resolve conflicts through the user defined conflict resolution settings. Name field conflicts will go through more extensive processing to resolve the conflicts. An estimate of the number of clusters containing conflicts and a count of clusters checked will be provided for users to gauge progress.

Users will be notified when the consolidation process is complete.

At times, the system may pause the working state and the subtitle of the step will be prefixed with the text System Paused.

The process of stopping consolidation and restoring the checkpoint goes through the following steps:

1. Stop
2. Restore Raw Entities
3. Restore Consolidated Entities

This step stops the consolidation process. If the stop action is issued while the consolidation action is queued, then the run consolidation is simply removed from the queue. Otherwise, if the consolidation process is in progress, it will be stopped and then the databases will be restored.

At times, the system may pause the working state and the subtitle of the step will be prefixed with the text System Paused.

At times, the system may pause the working state and the subtitle of the step will be prefixed with the text System Paused.

At times, the system may pause the working state and the subtitle of the step will be prefixed with the text System Paused.

Users will be notified for the following steps:

When the Entity Consolidation is picked up off the queue and synchronization starts, then the system will create a notification.

Notification

Notification

Notification

1. Sign in as a Project Administrator.
2. Select the project you are working on.
3. In the upper right corner of any page on Canopy, click on the Settings icon, then click on Activity under the Project tab.
   
4. Select the Class filter from the header of the data grid.
5. Add Consolidation to is any of selected filter.
6. Once filtered, scroll to your right and copy the Job ID of the run consolidation job you would like to filter.
7. Select the Job ID filter and paste the Job ID you just copied into the filter.
8. Paste the Job ID you just copied into the is any of these filter.
9. Filtering complete.