Index Data – Web
Indexing locations
In Reveal, there are multiple areas where you can index your data. Different areas provide different indexing functionalities.
Indexing in the Review Grid
Indexing in the Processing Grid – Processing Index is only metadata and OCR/Loading text sets. No other text sets apply here.
About Text Sets and Indexing
Discovery Manager processing: OCR / Loaded
OCR / Loaded is a default text set typically populated with one of two kinds of document text:
Extracted text from within the file.
Example: A Word file contains the exact textual contents written in its data, which can be extracted directly.
OCRed text
Example: A PDF file with no embedded text can be visually scanned by OCR to identify its characters. This may contain errors, depending on the quality of the image and of the OCR program.
Discovery Manager will process files in both ways. If the file has embedded text, it will extract it. If the file can be OCRed, Discovery Manager can run OCR. After Discovery Manager runs its processing engine, your files will either have:
No embedded or OCRed text – This occurs if the file errored during both kinds of processing (e.g. it didn’t have text in it, like an mp3 file).
Extracted text – OCR wasn’t possible (can’t be visually scanned).
OCRed text – There was no embedded text to extract.
OCRed and extracted text – Both processes were successful.
Discovery Manager can only choose to populate the OCR / Loaded text set with one of the two kinds of text. In the case where Discovery Manager successfully processed OCRed and embedded text, by default Discovery Manager will choose OCR over extracted. However, this setting can be changed in Discovery Manager to prioritize embedded text (during Export).
Reveal Web also provides the option to populate the OCR / Loaded text set during an OCR job run through the Review Grid. This will override any processed information done in Discovery Manager.
but it’s best practice to create your own custom OCR text set (see below) to supplement OCR / Loaded so that no critical pieces of data are lost from the original set.
Standard indexing workflows
There are three main ways to process data in Reveal. Each method has its own text sets that auto-index during processing, so further manual indexing is usually needed to generate all of Reveal’s pre-existing text sets for your data – Document_Metadata, Native View (PDF), Spreadsheet View, OCR / Loaded, Extracted, and Transcription.
Important
Rather than trying to generate Transcription or Spreadsheet text sets on your mixed data, it’s faster to isolate A/V files or Spreadsheet Files and perform separate Index jobs for each.
Method 1: Processing in Discovery Manager and Promote to Review
Discovery Manager
Create Import (Processing) job: This processes your data.
Create Review Append Export job: This will push your data from DM to Review, and auto-index the below text sets.
Document_Metadata, OCR / Loaded, Native View (PDF)
Reveal Web
Run an Index job: Select the below text sets to index.
Extracted
Isolate spreadsheet files: Use advanced search to search for just spreadsheet files.
Run a second Index job: Select the following text set to index for spreadsheets only.
Spreadsheet View
Isolate A/V files: Use advanced search to search for just A/V files.
Run a third Index job: Select the following text set for your A/V files only.
Transcription
Method 2: 3rd-party import in Review Manager
In Review Manager, process your 3rd-party data and load file.
No text sets are generated when processing 3rd-party data through Review Manager.
In Discovery Manager, run an Index job on all documents. This will auto-index the following text set:
Document_Metadata
In Reveal Web, run an advanced search that removes spreadsheets and A/V files.
Run an Index job on your remaining data. Select the following text sets to index:
OCR / Loaded, Extracted, Native View (PDF)
Isolate spreadsheet files through advanced search.
Run an Index job on your spreadsheet files. Select the following text sets to index:
OCR / Loaded, Extracted, Spreadsheet View
Isolate A/V files through advanced search.
Run an Index job on your A/V files. Select the following text set to index:
Transcription
Method 3: 3rd-party import in Reveal Web
In Reveal Web, process your 3rd-party data and load file. This will auto-index the following text sets:
Document_Metadata, OCR / Loaded, Spreadsheet View, Native View (PDF), Transcription
Run an Index job on your data. Select the following text set to index:
Extracted
Indexing Requirements article
Indexing requirements
Certain jobs in Reveal either require, automate, or allow the option to index text sets after the process has occurred.
Required indexing
If you’re performing an import job through Review Manager, metadata text sets require indexing.
Job Type | Required Indexing |
|---|---|
Import Job (Review Manager) | Metadata |
Automated indexing
Certain jobs will automatically perform indexing on specific text sets:
*OCR can be sent to the OCR / Loaded text set, or new text sets can be created to store the OCR.
Job Type | Auto-Indexing |
|---|---|
Import Job (Discovery Manager) | OCR / Loaded*, Metadata |
Import Job (Reveal Web) | OCR / Loaded*, Metadata |
OCR Job | OCR / Loaded*, Metadata |
Transcription Job | Transcription |
Translation Job | Translation |
Image Labeling Job | Metadata |
Note
Import jobs through Review Manager do not automatically index any text sets.
Optional indexing
Certain text sets can be indexed during the following jobs, but indexing is not required:
*OCR can be sent to the OCR / Loaded text set, or new text sets can be created to store the OCR.
Job Type | Optional Indexing |
|---|---|
Import Job (Discovery Manager) | Native View (PDF), Spreadsheet View, Extracted |
Import Job (Review Manager) | Native View (PDF), Spreadsheet View, Extracted, OCR / Loaded* |
Import Job (Reveal Web) | Native View (PDF), Spreadsheet View, Extracted |
OCR Job | Native View (PDF), Spreadsheet View |
Transcription Job | Native View (PDF), Spreadsheet View |
Translation Job | Native View (PDF), Spreadsheet View |
Color Detection Job | Native View (PDF), Spreadsheet View, Metadata |
Note
For Processing's "Review Append" exports (that is, native and other material ingested through Discovery Manager and not Review Manager), only the Document_Metadata & OCR Loaded will get indexed automatically through the API.
Items not initially indexed (other than Document Metadata, which must be indexed to populate the Grid) may be indexed in the Reveal Web environment. See Index Data – Web for more information.