Run a Hybrid GenAI Review

Overview

aji’s Hybrid GenAI Review is Reveal’s unique review mode that prioritizes speed and affordability. Faster and less expensive than GenAI Review, aji will apply GenAI ratings to small batches of documents and use the result of each batch to train a Classifier, influencing its rating ability against all your documents. When it reaches a certain defined condition (round number or precision / recall value), the Hybrid GenAI Review will stop.

How Does Hybrid GenAI Review Work?

Hybrid GenAI Review is a review process that’s defined by rounds. During each round:

• aji will batch a small amount of documents from your project. You can choose how many documents are batched each round.

• aji will review each document in the batch using GenAI, giving reasonings and citations for your documents.

• aji will use the rated documents to train the Classifier.

• The Classifier will score all of your documents in your project, giving each document a predictive score.

Hybrid GenAI Review will continue to run rounds until it meets a stopping condition. The default stopping condition is number of total rounds completed, which is a number you can customize to meet your review needs.

Important

Because there is only one Classifier associated and trained with each aji Workflow, all Hybrid GenAI Reviews for a workflow are cumulative, even if you edit or change your Definition or Context. See the Round Ranges section for more information.

Review Stopping Conditions

Each Hybrid GenAI Review will always stop once it reaches the total number of rounds completed, but you can further customize your review by adding any of the advanced stopping conditions, below:

• Stop after reviewing all documents above a specific predictive score.

• Stop after reaching a specific precision value.

• Stop after reaching a specific recall value.

• Stop after reaching both a specific precision value and recall value.

If your advanced stopping condition doesn’t get met by the time Hybrid GenAI Review meets your number of rounds completed, the review will still stop. Consider the advanced stopping condition as a way to end your review earlier than your total number of rounds.

Note

In order to use a precision or recall stopping condition, you must create a Control Set, then code all the documents in the Control Set.

Round Ranges

Each aji Workflow is associated with one Classifier used for Hybrid GenAI Reviews. Your first Hybrid review will use your current Definition and Context to train the Classifier and further fine-tune its ability to give your documents a predictive score. Any subsequent Hybrid reviews in that same workflow will be a continuation of prior training, even if you edit or completely change your Definitions or Context.

The below table shows an example of how one may change Definitions and Context in a single aji Workflow during their Hybrid GenAI Reviews.

Important

Once a document is given a GenAI rating in one of your Hybrid GenAI rounds, it will not be re-analyzed and given a new GenAI rating during other Hybrid GenAI rounds, even if you’ve changed your definition or context.

If you would like to start a fresh Hybrid GenAI Review on a new Classifier, you will need to create a new aji Workflow – there is no option to reset or delete your past Hybrid GenAI Reviews, or start again from Round 1 with a different Classifier.

Hybrid GenAI Review Steps

Hybrid GenAI Review will generally follow the below steps:

1. Create Your Control Set – Create a Control Set from the Review Grid.

2. Review Your Control Set – Code all the documents in your Control Set.

3. Start Hybrid GenAI Review – Perform Hybrid GenAI Review and choose your stopping conditions.

4. Interpret Hybrid GenAI Review Results – View the results of your Hybrid GenAI Review and determine next steps.

I. Create Your Control Set

Creating a Control Set allows you to select precision or recall values as stopping conditions. If you don’t want precision or recall stopping conditions, you can skip to step III, Start Hybrid GenAI Review.

1. Create your Control Set by following the instructions in our Create a Control Set article.

   1. Make sure you use the right Classifier and Tag Profile associated with your aji Workflow.

      1. To find the Classifier used, click the Classifier ID number in the Review Information pane (see “23”, below). This will open up the Classifier in the Supervised Learning screen.

         

      2. Choose the Tag Profile that contains your AI Tag. Your AI Tag can be identified in the Manually Reviewed pane (see Calibrate aji for more information on coding documents for aji reviews).

II. Review Your Control Set

1. Find your Control Set under the Assignments folder.

   • Your folder will have the naming convention [Classifier]_Control Set[BatchID], for example “HR-Payroll_Control Set00001”.

2. Code all documents in the folder using the correct AI Tag.

   • In order for the Classifier to work with your Control Set, you need to have coded at least 4 positive documents and 1 negative document in the set.

III. Start Hybrid GenAI Review

1. From the Hybrid GenAI Reviews module, click Add. The Hybrid GenAI Review modal will appear.

2. Name: Name your Hybrid GenAI Review.

   • Make sure you use a naming convention that makes sense to you and can help with easy reference across all your reviews.

3. Review Mode: This selection will be locked on Hybrid (Classifier + GenAI Review).

4. Rounds Completed: Choose the total number of rounds (cumulative number) you want to reach for your Hybrid GenAI Reviews.

   • For example, if you’ve already performed 20 Hybrid GenAI rounds in your aji Workflow, and you want your next Hybrid GenAI Review to consist of 15 rounds, you should type 35 into the Rounds Completed field to perform rounds 16–35. See Round Ranges for more information on how round numbers work for Hybrid GenAI Reviews.

5. Documents reviewed per round: Choose the number of documents the GenAI will review each round.

6. Advanced Options: Click Show Advanced Options to reveal advanced stopping conditions, which will stop the review even if you haven’t completed the round requirements.

   1. Reviewed All Docs Above Score: Choose a predictive score value. The review will stop if it reviews all documents above your set score.

   2. Classifier Performance:

      1. Recall Only – Stop after reaching a specific recall value.

         1. Min Recall Rate: Choose your recall rate.

         2. Using Score Threshold: Choose your score threshold.

      2. Precision Only – Stop after reaching a specific precision value.

         1. Min Precision Rate: Choose your precision rate.

         2. Using Score Threshold: Choose your score threshold.

      3. Both – Stop after reaching a specific Precision and Recall value. Both values must be met in order for the review to stop.

         1. Min Recall Rate: Choose your recall rate.

         2. Min Precision Rate: Choose your precision rate.

         3. Using Score Threshold: Choose your score threshold.

7. Click START REVIEW.

   • If you don’t have enough aji Units left in your project to start your GenAI Review, it won’t start and you will be notified via a popup. See our aji Unit Allocation – Instance Admin article for more details on assigning aji Units to projects, and our aji Billing article for details on how to estimate your aji Unit usage for a single Calibration, GenAI, or Hybrid GenAI Review.

Note

Additional information about Classifiers and score thresholds can be found in our Supervised Learning Overview article.

IV. Interpret Hybrid GenAI Review Results

There are two main tabs to view data for your Hybrid GenAI Reviews.

• The Performance tab displays the Score Distribution graph and Rounds table across all your Hybrid GenAI Reviews

• The Hybrid Run List tab lists individual Hybrid GenAI Reviews, lets you view GenAI ratings for the small batch of documents used to train the Classifier, and which Definition and Context was associated with your review.

Performance Tab

All data represented in the Performance data visualizations tab (the Score Distribution graph and Rounds table) is representative of all rounds across your Hybrid GenAI Reviews.

Score Distribution Graph

The Score Distribution graph contains the same information present in the Tagging & Scoring and Control Set tabs viewed after using a Classifier in Supervised Learning. The graph itself is the distribution of document predictive scores rated by the Classifier, which represents the Classifier’s predicted likelihood (0–100%) that the document is relevant to your Definition. A more elaborate explanation of the data in this tab can be found in our Evaluate an AI Model article.

Tagging & Scoring information is presented in the graph, and the total number of positive, negative, and not tagged documents are on to the right of the graph. You can also see how many documents are in your control set, and your total number of documents (all documents) in the review.

The Control Set information is to the bottom right, which provides precision, recall, F1, and richness values.

Just underneath the graph, the blue slider lets you move between round numbers to see how score distribution changed over time.

Rounds Table

The Rounds Table provides round-by-round metrics across your entire Hybrid GenAI Review. Above the table, you can find the AI Batch Size, or the number you put for the Documents reviewed per round field in the Hybrid GenAI Review modal, and the Max Rounds, or the total number of rounds completed across all Hybrid GenAI Reviews.

The table itself lists two sets of data: data related to the GenAI Ratings and data related to the Classifier’s predictive scores (Classifier Score, Recall, Precision, and F1).

Data related to your Classifier is dependent on a score threshold. The score threshold sets the line between which predictive score values are considered relevant (Yes) vs. not relevant (No). For example, a score threshold of 80 means all documents with a predictive score of 80 or above are considered relevant, and documents with a predictive score of 79 or lower are considered not relevant.

You can change your score threshold using the blue slider, or by entering a whole number value in the text box. Since score threshold influences precision, recall, and F1 calculations, changing the slider dynamically updates all the Classifier values in the table.

In the table, each column represents the following data:

• Round: The round number.

• GenAI Rating, Yes: Number of documents rated as “Strong Yes” or “Yes”.

• GenAI Rating, No: Number of documents rated as “Borderline” or “No”.

• Classifier Recall: The recall value of that round, considering your score threshold.

• Classifier Precision: The precision value of that round, considering your score threshold.

• Classifier F1: The F1 value of that round, considering your score threshold.

• Classifier Score ≥ X: The amount of documents equal to or above your score threshold.

• Classifier Score < X: The amount of documents less than your score threshold.

The full Rounds Table can be downloaded as a spreadsheet file by pressing the download icon in the upper right hand corner of the panel. Keep in mind that you will be downloading only the data for your chosen score threshold number from the slider.

Hybrid Run List Tab

The Hybrid Run List tab lets you select an individual Hybrid GenAI Review to view GenAI ratings per individual document, and the Definition and Context associated with that review. Notably, the Round range for run column shows you which rounds are associated with which Hybrid GenAI Review.

Documents Tab

The Documents tab for Hybrid GenAI Reviews functions the same as the Documents tab in Calibration and GenAI Reviews. You can view the GenAI ratings and open up documents to read reasonings and citations. See Calibrate aji for more information on this tab.

Note

The only documents visible in the Documents table are ones sent to the GenAI for review each round. In the Hybrid GenAI Review modal, you specified how many documents you would send for review by entering a number in the Documents reviewed per round field.