Auto Review

Welcome to Canopy’s Auto Review Feature! This guide will walk you through the key aspects of Canopy’s Auto Review feature, explaining how it works, how to set it up, and how to access the review results in Canopy. This feature is only available for projects created in version 5.0.0 or later.

Canopy’s Auto Review is a leading Agentic AI review software that uses AI and user-input Entity Qualifying Rules to automatically identify and propagate entities directly to the documents, designed to revolutionize your document review workflow.

Auto Review acts as the initial, first-level reviewer, streamlining your document review process by eliminating the need for manual human review on large and structured document sets. This automation dramatically increases efficiency and reduces costs.

Canopy supports Auto Review for most file types supported for Processing and hosting in Canopy, with the following exception:

1. Native Audio & Video files: Native audio & video files are not supported for Auto Review; however, transcribed audio & video are. Therefore, to include Audio & Video files in Auto Review, you must first transcribe them using Canopy’s Audio Transcription engine.
2. Database: All database files are not eligible for Auto Review

For more information on Canopy’s supported and unsupported file, please refer to File Types documentation.

A document can only go through Auto Review once; therefore, previously auto-reviewed documents are not eligible for another Auto Review Job within the same project.

For effective Auto Review, users must complete three essential steps to define the scope and criteria for the AI engine before initiating an Auto Review process.

To ensure highly accurate PII detection that aligns with your specific review workflow, you must select which PII elements are in scope for your project.

Required Action: Before process data into Canopy, the Data Processor must select the required PII Elements from the Processing Template, specifically under the PII Detection Options. Canopy provides a wide variety of PII options. Users can also upload custom detection rules to accommodate their unique detection patterns.

This step ensures the Canopy AI engine effectively detects and highlights the necessary PII elements with high accuracy, which the Auto Review engine will then use.

1. Navigate to Settings and select Template (under the Project settings).
2. In the Processing Templates section, click Manage processing template.

To ensure a standardized and consistent entity input during document view, Project Admins or Review Managers must create an Entity Layout for their review phase.

Entity Layout is a crucial display and organization tool for the key PIIs that you are trying to identify and review. It defines the exact data fields (i.e. First Name, Last Name, Social Security Number, Passport Number) that reviewers must look for or confirm during document review.

An Entity Layout is mandatory for every Auto Review. When creating an Auto Review Phase, users must select an Entity Layout. The layout is required because it tells the Auto Review AI engine precisely how many and where to map and propagate the detected PII into an entity.

1. Navigate to Settings and select Template (under the Project settings).
2. In the Entity Panel Layout section, click Manage entity layout.

3. Click Add New
4. Configure Fields:
   • Choose standards PII Elements from the System Fields,
   • To add a custom, non-system field, click Add Fields.
5. After configuring all required fields, click Save to create the Entity Layout

If you choose to add a custom field:

1. Select the desired Field Type and drag it toward the Layout Preview.
2. A sidebar pop-up will appear prompting you to define the new field. Complete the necessary information. For Auto Review entity layout, pay special attention to the following details:

Definition & Instruction

This is the complete definition of the custom field. In addition to definition, include step-by-step guidance for both human and AI reviewers on value identification and required entry conditions.

Keywords

This option asks users to enter up to 10 keywords that the human and AI reviewers should expect to see near the targeted key value (fields). These keywords help the AI engine precisely locate and confirm the custom field within the document text.

Example Values

This option ask users to enter up to 10 sample values that accurately mimic production data. All values entered here must be fabricated. They must not be derived from any real production data.

To ensure the AI engine accurately and consistently propagates entities to the auto reviewed documents, Project Admins or Review Managers must define and add Qualifying Rules.

Qualifying Rules are the mandatory, entity-defined criteria that determine how many and what kind of PII elements must exist in a document for it to be identified as a relevant “Entity” by the AI.

At least one Qualifying Rule is required for every Auto Review. Users must create and add this rule when creating the Auto Review phase.

Qualifying Rules are built using two field types: Primary Fields and Secondary Fields.

Primary Fields represent the minimum number of PII Elements that must be found within a document for the detected PIIs to be classified and considered as a complete Entity.

For example, your team requires an Entity to have a First Name, Last Name, and Social Security Number (SSN). Input the following PII elements as Primary Fields:

• First Name
• Last Name
• Social Security Number (SSN)

With this rule, the Auto Review engine will only propagate an Entity if the document contains all three of these PII elements.

Secondary Fields are supplementary PII elements used in conjunction with Primary Fields to validate a detected individual as a complete Entity. For a document to be flagged by the Auto Review engine, it must contain the required Primary Fields plus at least one of the defined Secondary Fields.

For example, in addition to Primary Fields listed above, your team also tracks Passport Number and Phone Number. Input the following PII as Secondary Fields:

• Passport Number
• Phone Number

The Auto Review engine applies the following logics:

• If the document contains the Primary Fields and contains both the Passport Number and Phone Number, the Auto Review engine will include the individual as an Entity.
• If the document contains the Primary Fields and contains either Passport Number or Phone Number, the Auto Review engine will include the individual as an Entity.
• If the document contains the Primary Fields but contains neither the Passport Number nor the Phone Number, the Auto Review engine will not include the individual as an Entity.

If multiple Secondary Fields are found for the same individual (as defined by the Primary Fields), our AI engine will propagate the reviewing document as having multiple entities.

For example: In a single document, AI engine finds the following information associated with the same individual (First Name: John, Last Name: Doe, SSN: 123-45_6789, US Passport: 123456789, EU Passport: AB123456), then our AI engine will propagate this information as 2 entities:

1. Entity 1: First Name: John, Last Name: Doe, SSN: 123-45_6789, US Passport: 123456789,
2. Entity 2: First Name: John, Last Name: Doe, SSN: 123-45_6789, EU Passport: AB123456.

Follow these steps to execute an Auto Review:

A Review Phase serves as a container for a specific set of documents that users intend to have reviewed. Canopy offers two kinds of Review Phase: Auto Review and Manual Review. For more information on Review Phase, refer to the Review Phase documentation.

Auto Review Phase is a container of documents intended for Auto Review.

1. Navigate to the Documents Page
2. Cull and Filter: Use Canopy’s smart search and filters to identify and narrow down the documents you wish to include in the Auto Review.
3. Select Documents: Select one or more documents to add to the review phase.
4. Access Menu: After documents have been selected, navigate to and click the kabab menu located in the top-right corner of the Documents page.
5. Add to Review Phase: Click on Review Phase and then select Add to Review Phase. The Add to Review Phase button will only be enabled if documents are selected on the Documents page. If no files are selected, the button will remain disabled.

6. Complete required information: You will be taken to the Add to Review Phase, and you will be prompt to stay in the Create Review Phase.

• Select Auto Review under Method
• Review under Type will be automatically selected for you.
• Entity Layout is required for Auto Review. If you do not have an entity layout yet, you can choose Click here to create a new layout.

Users must add at least one entity qualifying rule. Once the review phase is created, users cannot edit the Qualifying Rules; therefore, ensure that information are accurate before proceeding to save and create the review phase.

7. Once you’ve completed all the required information for your review phase, click Save to create it.

Saving at this stage does not trigger the Auto Review process. It only creates the container (the Review Phase).

8. The Review Phase Creation will begin as a background task. You will receive a system notification that the job has started.
9. Upon completion, you will receive a notification with two options:
   • Clicking Activity History takes you to the Activity History page, where you can view details of the completed job.
   • Clicking Batch List takes you to the Batches page where the newly created Review Phase is listed.

Auto Review Phases cannot be deleted or split once created. Please thoroughly verify all details of your Auto Review Phase before saving.

When you create an Auto Review Phase, Canopy automatically skips any documents that are not eligible for Auto Review. A document is ineligible if it:

• Has been previously auto reviewed (a document can only be reviewed once),
• Is a database file,
• Or already belong to another Auto Review Phase (a document cannot be included in multiple Auto Review Phases).

After an Auto Review Phase is created, users can Add or Remove documents from the newly created or existing Auto Review Phase. This condition applies only if the Auto Review Phase has not yet undergone the Auto Review process.

Once the Auto Review has completed, the Review Phase is locked. Users will no longer be able to Add or Remove any documents from the Auto Review Phase.

Refer to Review Phase documentation to learn how to Add or Remove documents from a Review Phase.

Once the Auto Review Phase is created, you must explicitly start the Auto Review process. Only Tenant Admin, Project Admin, and Review Manager can execute the Auto Review job.

11. Locate the Review Phase: On the Batches page, locate the Auto Review Phase you wish to run.
12. Start Auto Review: In the Action column for that review phase, click the Run button. This Run button is available only for the Auto Review Phase.

13. Confirm: A confirmation dialog will appear, displaying the total number of documents batched for Auto Review. Users must Click Auto Review to confirm and initiate the process.

14. Job Starts: The Auto Review will begin as a background task. You’ll receive a notification that the Auto Review has started.

15. Monitor Progress: You can track the Auto Review job’s progress in the Job Manager under the In Progress tab. Each job will be listed with a Job ID and current Status. Click on the Job ID for detailed information on the Auto Review job.

16. Job Completes: After the Auto Review is complete, you will receive a notification with options:

• Clicking Activity History takes you to the Activity History page to view details of the completed job.
• Clicking Document List takes you directly to the Documents page, filtered to show the newly Auto Reviewed files.

Canopy provides several intuitive method for efficiently finding and working with files that have completed the auto review process.

Completed Auto Review jobs are accessible through the Job Manager under the Completed tab.

1. Navigate to the Job Manager on the navigation bar.
2. Click the Completed tab to view all completed Auto Review jobs.
   • Click the Job ID to access the detailed Activity History log,
   • Or click View Results to access the Documents page, filtered to show the job’s auto reviewed documents.

Job entries in the Job Manager are automatically removed after 72 hours. Please review and access job results promptly.

Users can use the Batches page to find the Auto Reviewed documents by review phase.

1. Navigate to the Batches page.
2. Locate and click the desired Auto Review Phase to open it.
3. In the detail view, if the Auto Review job is complete, the associated batch will be found under the Completed tab. Select the Completed tab.
4. On the batch, click Document List
5. Click Reviewed tab to see the list of all documents within that Auto Review Phase that have completed the Auto Review job.

Users can use various filter columns on the Document List to isolate successfully auto reviewed or failed files.

1. Add Columns: On the Document List, add the following columns to the data grid:

   • Review Method,
   • Review Phase,
   • Review Type,
   • Review Status,
   • Reviewed Date/Time,
   • Reviewer Info.

2. Filter for Auto Review Result:

   • Step 1: Select Auto Review from the Review Method.
   • Step 2: Use the Review Phase column filter to select the desired Auto Review phase.
     • To find successfully Auto Reviewed files, select Reviewed from the Review Status column.
     • To find files that fail Auto Review, select Failed from the Review Status column.

All Qualifying Entities identified by the AI are automatically propagated into the document’s Entity pane. Here, users can delete, edit or add more entities to the Entity pane if need be.

The Entity data displayed in the Document View is directly linked to the Raw Entities within the Entities Module. When the Auto Review job completes, all propagated entities are automatically added to the Raw Entities pool. Any subsequent additions, edits, or deletions made in the Document View are dynamically updated in the Entities Module.

When opening Auto Reviewed documents from the Document List, you will not see the final review status. This is because Canopy requires documents to be accessed through a review batch for active review actions.

After an Auto Review job is complete, Canopy strongly suggests that users rebatch the Auto Reviewed documents for a Quality Assurance (QA) review.

The QA workflow is an essential second-level human reviewer check that adds another layer of assurance to the entity results generated by the AI engine before finalizing the review process.

Below are steps to create a manual QA review phase:

1. Navigate to the Documents Page
2. Filter: Use the data grid columns to filter for Auto Reviewed documents you wish to include in the QA.
3. Select Documents: Select documents to add to the review phase.
4. Access Menu: Navigate to and click the kabab menu located in the top-right corner of the Documents page.
5. Add to Review Phase: Click on Review Phase and then select Add to Review Phase.
6. Complete required information:
   • Under Method, select Manual
   • Under Type, select QA
7. Once you’ve completed all the required information, click Save to create the Manual QA Review Phase.

Refer to the Review Phase documentation for details.

Auto Review is a paid feature in Canopy. The cost is determined by the total documents reviewed using the Auto Review feature.

Only Tenant Admins have access to the total monthly auto-reviewed documents for their tenant. This information is displayed in the Tenant Settings under the Billing section within the Active Project report.

For more information on the pricing, please contact Customer Success.