Overview
aji’s Hybrid GenAI Review is Reveal’s unique review mode that prioritizes speed and affordability. Faster and less expensive than GenAI Review, aji will apply GenAI ratings to small batches of documents and use the result of each batch to train a Classifier, influencing its reviewing ability against all your documents. When it reaches a certain defined condition (round number or precision / recall value), the Hybrid GenAI Review will stop.
How Does Hybrid GenAI Review Work?
Hybrid GenAI Review is a review process that’s defined by rounds. During each round:
aji will batch a small amount of documents from your project. You can choose how many documents are batched each round.
aji will review each document in the batch using GenAI, giving reasonings and citations for your documents.
aji will use the rated documents to train the Classifier.
The Classifier will score all of your documents in your project, giving each document a predictive score.
Hybrid GenAI Review will continue to run rounds until it meets a stopping condition. The default stopping condition is number of rounds completed, which is a number you can customize to meet your review needs.
Each Hybrid GenAI Review will always have a number of rounds completed as a stopping condition, but you can further customize your review by adding any of the advanced stopping conditions, below:
Stop after reviewing all documents above a specific predictive score.
Stop after reaching a specific precision value.
Stop after reaching a specific recall value.
Stop after reaching both a specific precision value and recall value.
If your advanced stopping condition doesn’t get met by the time Hybrid GenAI Review meets your number of rounds completed, the review will still stop. Consider the advanced stopping condition as a way to end your review earlier than your total number of rounds.
Note
In order to use a precision or recall stopping condition, you must create a Control Set, then code all the documents in the Control Set.
Hybrid GenAI Review Steps
Hybrid GenAI Review will generally follow the below steps:
Create Your Control Set – Create a Control Set from the Review Grid.
Review Your Control Set – Code all the documents in your Control Set.
Start Hybrid GenAI Review – Perform Hybrid GenAI Review and choose your stopping conditions.
Interpret Hybrid GenAI Review Results – View the results of your Hybrid GenAI Review and determine next steps.
I. Create Your Control Set
Creating a Control Set allows you to select precision or recall values as stopping conditions. If you don’t want precision or recall stopping conditions, you can skip to step III, Start Hybrid GenAI Review.
Create your Control Set by following the instructions in our Create a Control Set article.
Make sure you use the right Classifier and Tag Profile associated with your aji Workflow.
To find the Classifier used, click the Classifier ID number in the Review Information pane (see “23”, below). This will open up the Classifier in the Supervised Learning screen.
.png)
Choose the Tag Profile that contains your AI Tag. Your AI Tag can be identified in the Manually Reviewed pane (see Calibrate aji for more information on coding documents for aji reviews).
II. Review Your Control Set
Find your Control Set under the Assignments folder.
Your folder will have the naming convention [Classifier]_Control Set[BatchID], for example “HR-Payroll_Control Set00001”.
Code all documents in the folder using the correct AI Tag.
In order for the Classifier to work with your Control Set, you need to have coded at least 4 positive documents and 1 negative document in the set.
III. Start Hybrid GenAI Review
From the Hybrid GenAI Reviews module, click Add. The Hybrid GenAI Review modal will appear.
Name: Name your Hybrid GenAI Review.
Make sure you use a naming convention that makes sense to you and can help with easy reference across all your reviews.
Review Mode: This selection will be locked on Hybrid (Classifier + GenAI Review).
Rounds Completed: Choose a number of rounds to complete before the review stops.
Documents reviewed per round: Choose a number of documents the GenAI will review each round.
Advanced Options: Click Show Advanced Options to reveal advanced stopping conditions, which will stop the review even if you haven’t completed the round requirements.
Reviewed All Docs Above Score: Choose a predictive score value. The review will stop if it reviews all documents above your set score.
Classifier Performance:
Recall Only – Stop after reaching a specific recall value.
Min Recall Rate: Choose your recall rate.
Using Score Threshold: Choose your score threshold.
Precision Only – Stop after reaching a specific precision value.
Min Precision Rate: Choose your precision rate.
Using Score Threshold: Choose your score threshold.
Both – Stop after reaching a specific Precision and Recall value. Both values must be met in order for the review to stop.
Min Recall Rate: Choose your recall rate.
Min Precision Rate: Choose your precision rate.
Using Score Threshold: Choose your score threshold.
Click START REVIEW.
If you don’t have enough aji Units left in your project to start your GenAI Review, it won’t start and you will be notified via a popup. See our aji Unit Allocation – Instance Admin article for more details on assigning aji Units to projects, and our aji Billing article for details on how to estimate your aji Unit usage for a single Calibration, GenAI, or Hybrid GenAI Review.
Note
Additional information about Classifiers and score thresholds can be found in our Supervised Learning Overview article.
IV. Interpret Hybrid GenAI Review Results
There are three main data visualization tabs for Hybrid GenAI Review: Results, Documents, and Rounds.
Hybrid GenAI Review Data Visualizations
Results Tab
The Results tab functions similarly to the Tagging & Scoring and Control Set tabs viewed after using a Classifier in Supervised Learning. The graph itself is the distribution of document predictive scores rated by the Classifier, which represents the Classifier’s predicted likelihood (0–100%) that the document is relevant to your Definition. A more elaborate explanation of the data in this tab can be found in our Evaluate an AI Model article.
Tagging & Scoring information is presented in the graph, and the total number of positive, negative, and not tagged documents are on to the right of the graph. You can also see how many documents are in your control set, and your total number of documents (all documents) in the review.
The Control Set information is to the bottom right, which provides precision, recall, F1, and richness values.
Just underneath the graph, the blue slider lets you move between round numbers to see how score distribution changed over time.
Documents Tab
The Documents tab functions the same as the Documents tab in Calibration and GenAI Review. You can view the GenAI ratings and open up documents to read reasonings and citations. See Calibrate aji for more information on this tab.
Note
The only documents visible in the Documents table are ones sent to the GenAI for review each round. In the Hybrid GenAI Review modal, you specified how many documents you would send for review by entering a number in the Documents reviewed per round field.
Rounds tab
The Rounds tab provides round-by-round metrics across your entire Hybrid GenAI Review. Above the table, you can find the AI Batch Size, or the number you put for the Documents reviewed per round field in the Hybrid GenAI Review modal, and the Max Rounds you set for the review as a stopping condition.
.png)
The table itself lists two sets of data: data related to the GenAI Ratings and data related to the Classifier’s predictive scores (Classifier Score, Recall, Precision, and F1).
Data related to your Classifier is dependent on a score threshold. The score threshold sets the line between which predictive score values are considered relevant (Yes) vs. not relevant (No). For example, a score threshold of 80 means all documents with a predictive score of 80 or above are considered relevant, and documents with a predictive score of 79 or lower are considered not relevant.
You can change your score threshold using the blue slider, or by entering a whole number value in the text box. Since score threshold influences precision, recall, and F1 calculations, changing the slider dynamically updates all the Classifier values in the table.
In the table, each column represents the following data:
Round: The round number.
GenAI Rating, Yes: Number of documents rated as “Strong Yes” or “Yes”.
GenAI Rating, No: Number of documents rated as “Borderline” or “No”.
Classifier Recall: The recall value of that round, considering your score threshold.
Classifier Precision: The precision value of that round, considering your score threshold.
Classifier F1: The F1 value of that round, considering your score threshold.
Classifier Score ≥ X: The amount of documents equal to or above your score threshold.
Classifier Score < X: The amount of documents less than your score threshold.
The full Rounds Table can be downloaded as a spreadsheet file by pressing the 
download icon in the upper right hand corner of the panel. Keep in mind that you will be downloading only the data for your chosen score threshold number from the slider.