Auto Review

Welcome to Canopy’s Auto Review Feature! This guide will walk you through the key aspects of Canopy’s Auto Review feature, explaining how it works, how to set it up, and how to access the review results in Canopy. This feature is only available for projects created in version 5.0.0 or later.

Canopy’s Auto Review is a leading Agentic AI review software that uses AI and user-input Entity Qualifying Rules to automatically identify and add entities directly to the documents, designed to revolutionize your document review workflow.

Auto Review acts as the initial, first-level reviewer, streamlining your document review process by eliminating the need for manual human review. This automation dramatically increases efficiency and reduces costs.

PII Validation Exclusion: PII Elements that fail format or checksum validation are considered invalid by the AI and will be excluded from the final Entity extraction and propagation.

Canopy supports Auto Review for most file types supported for Processing and hosting in Canopy, with the following exception:

1. Native Audio & Video files: Native audio & video files are not supported for Auto Review; however, transcribed audio & video are. Therefore, to include Audio & Video files in Auto Review, you must first transcribe them using Canopy’s Audio Transcription engine.
2. Database: All database files are not eligible for Auto Review

For more information on Canopy’s supported and unsupported file, please refer to File Types documentation.

A document can only go through Auto Review once; therefore, previously auto-reviewed documents are not eligible for another Auto Review Job within the same project.

For effective Auto Review, users must complete three essential steps to define the scope and criteria for the AI engine before initiating an Auto Review process.

To ensure highly accurate PII detection that aligns with your specific review workflow, you must select which PII elements are in scope for your project.

Required Action: Before process data into Canopy, the Data Processor must select the required PII Elements from the Processing Template, specifically under the PII Detection Options. Canopy provides a wide variety of PII options. Users can also upload custom detection rules to accommodate their unique detection patterns.

This step ensures the Canopy AI engine effectively detects and highlights the necessary PII elements with high accuracy, which the Auto Review engine will then use.

1. Navigate to Settings and select Template (under the Project settings).
2. In the Processing Templates section, click Manage processing template.
3. Choose the desired PII elements from the PII Detection Options.
4. Click on Save As New Template, and you will be prompted to name the template.
5. When you start uploading and processing documents, make sure to select this template under the Processing Template option.

To ensure a standardized and consistent entity input during document view, Project Admins or Review Managers must create an Entity Layout for their review phase.

Entity Layout is a crucial display and organization tool for the key PIIs that you are trying to identify and review. It defines the exact data fields (i.e. First Name, Last Name, Social Security Number, Passport Number) that reviewers must look for or confirm during document review.

An Entity Layout is mandatory for every Auto Review. When creating an Auto Review Phase, users must select an Entity Layout. The layout is required because it tells the Auto Review AI engine precisely how many and where to map the detected PII into an entity.

1. Navigate to Settings and select Template (under the Project settings).
2. In the Entity Panel Layout section, click Manage entity layout.

3. Click Add New
4. Configure Fields:
   • Choose standards PII Elements from the System Fields,
   • To add a custom, non-system field, click Add Fields.
5. After configuring all required fields, click Save to create the Entity Layout

If you choose to add a custom field:

1. Select the desired Field Type and drag it toward the Layout Preview.
2. A sidebar pop-up will appear prompting you to define the new field. Complete the necessary information. For Auto Review entity layout, pay special attention to the following details:

Definition & Instruction

This is the complete definition of the custom field. In addition to definition, include step-by-step guidance for both human and AI reviewers on value identification and required entry conditions.

Keywords

This option asks users to enter up to 10 keywords that the human and AI reviewers should expect to see near the targeted key value (fields). These keywords help the AI engine precisely locate and confirm the custom field within the document text.

Example Values

This option ask users to enter up to 10 sample values that accurately mimic production data. All values entered here must be fabricated. They must not be derived from any real production data.

Auto Review does not support Other Personal Info Checkbox custom fields at this time. To use these in your project, contact Customer Success to coordinate specialized support from the Canopy Data Science team.

To ensure the AI engine accurately and consistently add entities to the auto reviewed documents, Project Admins or Review Managers must define and add Qualifying Rules.

Qualifying Rules are the mandatory, entity-defined criteria that determine how many and what kind of PII elements must exist in a document for it to be identified as a relevant “Entity” by the AI.

At least one Qualifying Rule is required for every Auto Review. Users must create and add this rule when creating the Auto Review phase.

Users must add the entities’ Qualifying Rules when creating a new Auto Review Phase.

Qualifying Rules are built using two field types: Primary Fields and Secondary Fields.

Primary Fields represent the minimum number of PII Elements that must be found within a document for the detected PIIs to be classified and considered as a complete Entity.

For example, your team requires an Entity to have a First Name, Last Name, and Social Security Number (SSN). Input the following PII elements as Primary Fields:

• First Name
• Last Name
• Social Security Number (SSN)

With this rule, the Auto Review engine will only add an Entity if the document contains all three of these PII elements.

Secondary Fields are supplementary PII elements used in conjunction with Primary Fields to validate a detected individual as a complete Entity. For a document to be flagged by the Auto Review engine, it must contain the required Primary Fields plus at least one of the defined Secondary Fields.

For example, in addition to Primary Fields listed above, your team also tracks Passport Number and Phone Number. Input the following PII as Secondary Fields:

• Passport Number
• Phone Number

The Auto Review engine applies the following logics:

• If the document contains the Primary Fields and contains both the Passport Number and Phone Number, the Auto Review engine will include the individual as an Entity.
• If the document contains the Primary Fields and contains either Passport Number or Phone Number, the Auto Review engine will include the individual as an Entity.
• If the document contains the Primary Fields but contains neither the Passport Number nor the Phone Number, the Auto Review engine will not include the individual as an Entity.

If multiple Secondary Fields are found for the same individual (as defined by the Primary Fields), our AI engine will treat the reviewing document as having multiple entities.

For example, in a single document, AI engine finds the following information associated with the same individual (First Name: John, Last Name: Doe, SSN: 123-45_6789, US Passport: 123456789, EU Passport: AB123456), then our AI engine will add 2 entities:

1. Entity 1: First Name: John, Last Name: Doe, SSN: 123-45_6789, US Passport: 123456789,
2. Entity 2: First Name: John, Last Name: Doe, SSN: 123-45_6789, EU Passport: AB123456.

After Auto Review, the AI engine provides Suggested Entities when it identifies:

• Additional PII elements that fall outside the Qualifying Rules of the added entities,
• Potential entities that may not strictly meet the Qualifying Rules but contain significant PII.

Follow these steps to execute an Auto Review:

A Review Phase serves as a container for a specific set of documents that users intend to have reviewed. Canopy offers two kinds of Review Phase: Auto Review and Manual Review. For more information on Review Phase, refer to the Review Phase documentation.

Auto Review Phase is a container of documents intended for Auto Review.

1. Navigate to the Documents Page
2. Cull and Filter: Use Canopy’s smart search and filters to identify and narrow down the documents you wish to include in the Auto Review.
3. Select Documents: Select one or more documents to add to the review phase.
4. Access Menu: After documents have been selected, navigate to and click the kabab menu located in the top-right corner of the Documents page.
5. Add to Review Phase: Click on Review Phase and then select Add to Review Phase. The Add to Review Phase button will only be enabled if documents are selected on the Documents page. If no files are selected, the button will remain disabled.

6. Complete required information: You will be taken to the Add to Review Phase, and you will be prompt to stay in the Create Review Phase.

Review Information

• Select Auto Review under Method
• Review under Type will be automatically selected for you.
• Entity Layout is required for Auto Review. If you do not have an entity layout yet, you can choose Click here to create a new layout.

Qualifying Rules

Users must add at least one entity qualifying rule. Once the review phase is created, users cannot edit the Qualifying Rules; therefore, ensure that information are accurate before proceeding to save and create the review phase.

7. Once you’ve completed all the required information for your review phase, click Save to create it.

Saving at this stage does not trigger the Auto Review process. It only creates the container (the Review Phase).

8. The Review Phase Creation will begin as a background task. You will receive a system notification that the job has started.
9. Upon completion, you will receive a notification with two options:
   • Clicking Activity History takes you to the Activity History page, where you can view details of the completed job.
   • Clicking Batch List takes you to the Batches page where the newly created Review Phase is listed.

Auto Review Phases cannot be deleted or split once created. Please thoroughly verify all details of your Auto Review Phase before saving.

When you create a new Auto Review Phase, Canopy automatically skips any documents that are not eligible for Auto Review. A document is ineligible if it:

• Has been previously auto reviewed (a document can only be reviewed once),
• Is a database file,
• Is a native audio or video file,
• Or is pending Auto Review in another Auto Review Phase (a document cannot be included in multiple Auto Review Phases).

To find the skipped document, users have two options:

You can find documents skipped during the Auto Review batching process in the Project Activity History.

1. Navigate to the Project’s Activity History
2. Filter:
   • Set the Class filter to Review Phase Creation
   • Set the Job ID filter to your specific Auto Review Phase job.
   • This displays all statuses related to that specific Auto Review Phase creation.
3. View Skipped Documents: Set the Type filter to Skipped. This will display only the documents that were skipped when batching documents for the Auto Review Phase.
4. Access Documents: Click a Document ID within the Description column to open that document in the Document View.

You can also find skipped documents directly within the specific Auto Review Phase.

1. Navigate to the Batches Page
2. Select your desired Auto Review Phase from the list.
3. Click the Skipped tab.
4. In the Batch information, click the Document List to see all skipped documents and the specific reason they are excluded from Auto Review.

Once you’ve identified all the skipped documents, we recommend:

1. Manual Review: Re-batch the documents for human review.
2. Auto Review: Resolve the requirement (e.g., transcribe the audio & videos) and re-batch the documents into a different Auto Review Phase.

You can add more documents to an existing or newly created Auto Review Phase, provided the Auto Review process has not yet started.

Once the Auto Review is complete, the Review Phase is locked. Users can no longer Add any documents to that specific Auto Review Phase.

For detailed instructions on how to Add documents to a Review Phase, refer to Review Phase documentation.

When adding documents, ensure you select the correct Auto Review Phase from the Name list.

You can remove documents from an existing or newly created Auto Review Phase, provided the Auto Review process has not yet started.

Once the Auto Review is complete, the Review Phase is locked. Users can no longer Remove any documents from that specific Auto Review Phase.

For detailed instructions on how to Remove documents from a Review Phase, refer to Review Phase documentation.

When removing documents, ensure you select the correct Auto Review Phase from the Review Phase list if you choose the option to “Remove from specific review phase”.

Once the Auto Review Phase is created, you must explicitly start the Auto Review process. Only Tenant Admin, Project Admin, and Review Manager can execute the Auto Review job.

11. Locate the Auto Review Phase: On the Batches page, locate the Auto Review Phase you wish to run.
12. Start Auto Review: In the Action column for that review phase, click the Run button. This Run button is available only for the Auto Review Phase.

13. Confirm: A confirmation dialog will appear, displaying the total number of documents batched for Auto Review. Users must Click Auto Review to confirm and initiate the process.

14. Job Starts: The Auto Review will begin as a background task. You’ll receive a notification that the Auto Review has started.

15. Monitor Progress: You can track the Auto Review job’s progress in the Job Manager under the In Progress tab. Each job will be listed with a Job ID and current Status. Click on the Job ID for detailed information on the Auto Review job.

16. Job Completes: After the Auto Review is complete, you will receive a notification with options:

• Clicking Activity History takes you to the Activity History page to view details of the completed job.
• Clicking Document List takes you directly to the Documents page, filtered to show the newly Auto Reviewed files.

When users initiate an Auto Review process, Canopy automatically skips any documents that would cause the process to exceed the cost limits outlined in your Agreement. These documents are then redirected to a Skipped Documents status to prevent them from undergoing Auto Review.

You can view and manage these documents in the Skipped tab of the Batches module:

1. Navigate to the Batches Page
2. Select your desired Auto Review Phase from the list.
3. Click the Skipped tab.
4. In the Batch information, click the Document List to see all skipped documents and the specific reason they are excluded from Auto Review.
5. Check the “Reason” column for the message “Exceed Auto Review cost limits defined in the Agreement”.

Once identified, you have two options for reviewing these documents:

1. Manual Review: Re-batch the documents for human review.
2. Auto Review: Re-batch the documents into a different Auto Review Phase if the pricing threshold permits.

Canopy provides several intuitive method for efficiently finding and working with files that have completed the auto review process.

Completed Auto Review jobs are accessible through the Job Manager under the Completed tab.

1. Navigate to the Job Manager on the navigation bar.
2. Click the Completed tab to view all completed Auto Review jobs.
   • Click the Job ID to access the detailed Activity History log,
   • Or click View Results to access the Documents page, filtered to show the job’s auto reviewed documents.

Job entries in the Job Manager are automatically removed after 72 hours. Please review and access job results promptly.

Users can use the Batches page to find the Auto Reviewed documents by review phase.

1. Navigate to the Batches page.
2. Locate and click the desired Auto Review Phase to open it.
3. In the detail view, if the Auto Review job is complete, the associated batch will be found under the Completed tab. Select the Completed tab.
4. On the batch, click Document List
5. Click Reviewed tab to see the list of all documents within that Auto Review Phase that have completed the Auto Review job.

Users can use Review-related columns on the Document List to isolate successfully auto reviewed or failed files.

1. Add Columns: On the Document List, add the following columns to the data grid:
   • Review Method,
   • Review Phase,
   • Review Type,
   • Batch,
   • Review Status,
   • Reviewed Date/Time,
   • Reviewer Info.

2. Filter for Auto Review Result:

   • Step 1: Select Auto Review from the Review Method.
   • Step 2: Use the Review Phase column filter to select the desired Auto Review phase.
     • To find successfully Auto Reviewed files, select Reviewed from the Review Status column.
     • To find files that fail Auto Review, select Failed from the Review Status column.

Grouped Review-Related Columns

To simplify readability, we’ve grouped together Review-related column as a single functional group. When you modify the display settings for one of these columns, the entire group updates automatically.

• Visibility: Adjusting the visibility (Show more/Show less) for any Review-related field will cause the entire group to sync automatically.
• Data Alignment: As you read the Columns from left to right, the rows remain aligned.

The grouped columns include the seven columns mentioned above.

All Qualifying Entities identified by the AI are automatically added into the document’s Entity pane. Here, users can delete, edit or add more entities to the Entity pane if need be.

The Entity data displayed in the Document View is directly linked to the Raw Entities within the Entities Module. When the Auto Review job completes, all Auto Reviewed entities are automatically added to the Raw Entities pool. Any subsequent additions, edits, or deletions made in the Document View are dynamically updated in the Entities Module.

If Open from Inside the Batch

If Open from Document List

When opening Auto Reviewed documents from the Document List, you will not see the final review status. This is because Canopy requires documents to be accessed through a review batch for active review actions.

The Auto Review Confidence Level estimates the accuracy of the Auto Review results by combining three main components: the document’s Handwriting Score, Image Quality Score, and Entity Matching Score.

• Handwriting Score represents how well the handwriting matches known characters or words.
• Image Quality Score is determined by blur, contrast, and brightness.
• The Entity Matching Score is the average accuracy of all entities added by Auto Review, measuring how closely each element within the entity matches the document’s text.

Canopy categorizes confidence into three distinct levels:

• High (> 80%)
• Medium (50% – 80%)
• Low (< 50%)

Text-based files (such as .txt, .docx, .eml, and more) will currently default to Low Confidence if the Auto Review does not add any entity, regardless of the document’s actual content.

Columnar files (such as spreadsheet, Excel, .csv, and more) will default to High Confidence.

Users can view and manage the confidence levels directly from the Document List data grid:

1. Navigate: Open the Documents Module.
2. Add Columns: On the Document List, add Auto Review Confidence Level to the data grid.
3. Filter: Use the column filter to isolate specific Confidence Level (e.g., filtering for “Low” to identify documents requiring immediate manual review).

Consistent with standard manual review practices, we recommend that all documents undergo Quality Assurance (QA).

Below are steps to create a manual QA review phase:

1. Navigate to the Documents Page
2. Filter: Use the data grid columns to filter for Auto Reviewed documents you wish to include in the QA.
3. Select Documents: Select documents to add to the review phase.
4. Access Menu: Navigate to and click the kabab menu located in the top-right corner of the Documents page.
5. Add to Review Phase: Click on Review Phase and then select Add to Review Phase.
6. Complete required information:
   • Under Method, select Manual
   • Under Type, select QA
7. Once you’ve completed all the required information, click Save to create the Manual QA Review Phase.

Refer to the Review Phase documentation for details.

Auto Review is a paid feature in Canopy. The cost is determined by the total documents reviewed using the Auto Review feature.

Only Tenant Admins have access to the total monthly auto-reviewed documents for their tenant. This information is displayed in the Tenant Settings under the Billing section within the Active Project report.

For more information on the pricing, please contact Customer Success.