Early Access
This article has been published prior to aji’s GA date. Content may be subject to updates before its official release.
Overview
The Calibration stage is an iterative process of sending documents to GenAI for review; comparing those ratings to manual, human review; then refining Definitions to improve aji’s rating accuracy. Your aim is to improve your Definitions and repeat Calibration Runs to increase your Agreement Rate, or the percentage of documents aji rates that match your own coding decisions.
Calibration is performed by following the below steps:
Start a Calibration Run – Decide where to source your documents, and how many to run.
Code Documents – Code the documents used in the Calibration Run.
Interpret Calibration Results – After Calibration completes, assess your Agreement Rate, and read aji’s reasonings and citations for its ratings.
Apply Calibration Findings – If you want to improve your Agreement Rate, edit your Definitions and perform another Calibration Run.
I. Start a Calibration Run
Calibration Module
Calibration Runs are created in the Calibration module, accessible in the left navigation pane. To start a new Calibration Run, press the ADD button.
.png)
The Agreement Rate Graph plots your Agreement Rate across all your Calibrations.
Your Calibration Run List contains all of your old Calibration Runs in your AI Workflow. You can click the name of a Calibration Run to see it’s individual results in the Calibration Data Visualizations, discussed further in the Interpret Calibration Results section.
Run a Calibration Test Modal
Note
For a quicker way to generate a document set for Calibration, choose Suggest documents automatically. This automatically runs your Definition(s) as an ASK search query, then creates a document set from the results of the ASK search.
From the Calibration module, press the ADD button. The Run a Calibration Test modal will appear.
Name This Run: Name your Calibration Run.
Make sure you use a naming convention that makes sense to you and can help with easy reference across all your runs.
Documents to Review: Select which documents you want aji to send to GenAI for review.
Suggest documents automatically – have aji generate its own set of documents for review. aji will put your Definition(s) through ASK as a search prompt, then use a select number of those documents.
Number of Documents to Select: Type the number of documents you want to use for the run.
Use existing document selection – use documents from a previous run.
From the dropdown, choose which aji run you want to reuse documents from.
Create new document selection – use a new set of documents from a Saved Search or Work Folder.
All Documents – Calibrate all documents in your project. This option is not recommended, since the Calibration stage is designed to test a small number of documents from your entire dataset.
Saved Search – Pull your documents from a pre-existing saved search. The Saved Searches modal will appear.
.png)
Choose a saved search. You can click the > arrow to expand and view the saved search as it would appear in the Review Grid search bar.
Click SELECT.
Work Folder – Pull your documents from a pre-existing work folder.
Use the > arrow to find and then choose a work folder.
Definition Suggestions: Check whether or not you would like aji to suggest Definition improvements based on the Calibration results.
See the Apply Calibration Findings section for information on where to find aji’s suggested Definition improvement.
Click RUN to start the Calibration process.
II. Code Documents
Review Documents
The Calibration Run is dependent on a human to manually review documents and compare them to aji’s results. To manually code your documents, return to the work folder or saved search used in your Calibration Run. If you chose to have aji Select documents automatically, you can find them in an auto-created work folder titled AI Calibration Sets.
.png)
Each aji Workflow will have it’s own designated work folder, titled with the name you provided in the Workflow Name field when you created your workflow. Documents selected automatically by aji can be identified by the number of documents generated in the Calibration Run and the timestamp it was created.
When you’re tagging documents in Document Viewer, make sure you’re using the correct AI tag associated your Calibration Run, which can be identified at the bottom of the Manually Reviewed pane from the Calibration Data Visualizations (e.g. “Tag documents with the tag HR-PAYROLL [w34]”).
Important
Your AI Tag represents your entire Definition Set, which could be composed of up to 10 Definitions. As you’re coding your documents, you should be marking them as responsive if they meet at least one of your Definitions in the entire Definition Set.
Tag Sync
Once you have finished coding your documents, they must go through a Tag Sync to show up in aji’s data visualizations, which is performed automatically by Reveal. Tag Syncs can take anywhere from 15 minutes to around 1 hour. The Manually Reviewed and Evaluation sections will populate after tags successfully sync in the system.
III. Interpret Calibration Results
Once you’ve started your Calibration Run, you can view it by selecting its name from the Calibration Run List table.
After the Calibration Run completes, only a portion of the Calibration Data Visualizations will load, shown below. The rest of the visualizations will populate upon receiving the manually coded data.
.png)
Important
As you’re coding documents, Reveal’s automatic Tag Sync process will update your manually reviewed documents with the data visualizations on a rolling basis. This means the data visualizations will update with data—including the Agreement Rate—even if you don’t finish coding all documents.
Note that these visualizations are best interpreted after you have finished reviewing all documents used in your Calibration Run.
Run Information Pane
At the top of your data visualizations, you can see information for your current run (Calibration, Hybrid Review, or Full Review) and information about your aji Workflow.
Date Created: The date your current run was created.
Last Run: The date your latest run was performed.
Classifier: The Classifier associated with your aji Workflow, used in Hybrid Runs.
Status: The status of your current run.
Score Sync: The status of your Score Sync, which syncs aji’s ratings across the entire Reveal system (aji, Review Grid, Document Viewer, etc.).
This is different from a Tag Sync, which syncs your manually reviewed coding with aji’s data visualizations.
Run By: Who initiated the current run.
You can also create new runs from this pane.
Review: Start a Hybrid Review or Full Review.
Calibrate: Start another Calibration Run.
Calibration Results tab
The Calibration Results tab is the central hub for your Calibration Run results. Two donut wheels represent data: GenAI Reviewed contains aji’s ratings, and Manually Reviewed contains your coding decisions. Hover your mouse over either donut wheel to see the value of that section of data.
While your manual coding is decided with either a “yes” (relevant) or “no” (not relevant), aji’s ratings follow a more precise scale:
No – The document show very little or no alignment with the Definition.
Borderline – The document shows partial alignment with the Definition. Some elements are relevant, but important aspects are missing, unclear, or inconsistent, making the overall rating uncertain.
Yes – The document meets the Definition with sufficient relevance. While it may not cover every nuance, it clearly reflects what reviewers are looking for.
Strong Yes – The document strongly aligns with the Definition. It addresses key criteria in a clear, accurate, and comprehensive manner.
Documents that aji has an issue reading or interpreting will be marked Not Analyzed (rating) and Not Available (donut wheel).
Note
If you created multiple Definitions in your Workflow (collectively called a Definition Set), then aji will evaluate each individual Definition with one of the decisions in the above rating scale for all documents. When it comes time to rate the document, aji will favor the highest rated Definition (Strong Yes > Yes > Borderline > No) in the Definition Set, and that one rating will be represented in the data visualizations.
In the Agreement Report by Document tab, you can select a single document to see a breakdown of aji’s decisions for each Definition in your Definition Set.
Evaluation pane
The Evaluation pane lists the Calibration Run’s metrics, comparing your and aji’s results to provide important feedback about aji’s accuracy when rating documents. The Agreement Rate can be found here.
There are four possible agreement outcomes between your and aji’s reviews. Note that, for the purpose of determining agreement, “Borderline” documents are considered a “no”.
Agreement | Your Rating | aji’s Rating |
|---|---|---|
True Positive | Yes | “Strong Yes” or “Yes” |
True Negative | No | “Borderline” or “No” |
False Positive | No | “Strong Yes” or “Yes” |
False Negative | Yes | “Borderline” or “No” |
All four outcomes are listed in the evaluation pane with the number of documents for each outcome.
Manual Report tab
The Manual Report is a contingency table that reports document numbers across your coding and aji’s ratings. If there are documents you have yet to code, they’ll appear in the “Not Labelled” row.
Agreement Report by Document tab
The Agreement Report by Document tab is where you go to investigate aji’s ratings, reading its generated reasoning and citations to understand how well it responds to your Definition.
.png)
The rightmost columns list the GenAI Rating (aji’s ratings), Manual Decision (your coding decisions), and the Agreement outcome. To the left, you can click any of the BegDoc numbers to open up the Document Preview pane and take a closer look at aji’s ratings.
From this tab, you can also perform the following actions:
Open up one of your documents in Document Viewer, in a new browser. 
Filter the table by GenAI Rating, Manual Decision, or Agreement.
Create a search in the Review Grid containing all the documents from your Calibration Run, which can be saved.
Download the Agreement Report as a spreadsheet file.
Document Preview
The Document Preview window lets you see individual Definitions and their aji rating, reasoning(s), and citations.
In the rightmost AI Review pane, the top two dropdowns let you change between any Workflows and Runs that the document is associated with. You can find your Definitions underneath these dropdowns, which can be expanded / collapsed using the > arrow to see aji’s reasonings.
Click any of the blue reference numbers in your reasoning to jump to a highlighted citation in your document. You can also copy the reasoning to your clipboard or download it as a text file.
.png)
Document content has been intentionally blurred in the above image.
The rightmost toolbar lets you change between three tabs:
.png)
Document Explorer, where you can search your document’s text for hits..png)
ASK, which lets you ask questions about your document’s content using GenAI.
AI Review, where you can see aji’s reasonings and citations for each Definition. This opens by default when entering Document Preview through the Agreement Report by Document tab.
IV. Apply Calibration Findings
After exploring all the tabs in the Calibration Data Visualizations, you can use the information you found to improve your Definitions, then see if the Agreement Rate increases with another Calibration Run. To edit your Definition or Context, you can navigate to the Definitions or Context tabs, then click the .png){kind=small}
pencil icon.
Auto-Tuned Definition
If you marked the checkbox for Definition Suggestions in the Run a Calibration Test modal, you can access the suggested Definition from the Run Information pane by clicking Auto-tuned Definition. Auto-tuned Definitions are suggestions only; you don’t need to change anything if you prefer your own Definition as is.
Definition and Context History
Any time you edit your Definition Set or Context, Reveal will save your previous version for your records. These can be viewed and referenced in the Definition History and Context History tables, respectively.
To view either table, navigate to the Definitions module or Context module from the upper left navigation pane.
Definition History
The Definition History table provides a variety of information about your Definition Sets, including the list of runs that used each Definition Set. Click the Definition Set’s name to see the individual Definitions from your prior save.
The full Definition History can be downloaded as a spreadsheet file by pressing the download icon.
Context History
.png)
The Context History table provides a variety of information about your AI Workflow context, including the list of runs that were associated with each context section. Clicking content in any of the three context columns—People & Roles, Organizations & Entities, or Additional Context—will open up the context from your prior save.
The full Context History can be downloaded as a spreadsheet file by pressing the download icon.
Perform Another Calibration Run
If you choose to edit your Definitions and rerun Calibration, make sure to select the same Calibration Set from your previous run so you can see the impact of your Definition changes.
.png)
Running Calibration on a previous Calibration Set in the above manner will create a new Calibration Set, containing your new ratings from the next run.