Index Data

After importing documents has completed, the data will need to be indexed. An index at the most basic level locates words imported into the project to enable searching and highlighting of search hits; however, Reveal runs additional processes during this time (i.e., conversion into html, text extraction and color detection).

Note

With the deployment of Reveal 2024.5, bulk updates and incremental analytics indexing will run in parallel without queuing or interrupting running processes. This will facilitate continuous improvement of Reveal without impact on workflow.

Below is a chart that indicates whether indexing is required for specific Text Sets after the corresponding event has occurred.

Note

Image Labeling, like Transcription and Translation, kicks off its own indexing job for the metadata generated.

Text Set descriptions are:

• HTML/Native - Extracted HTML content from native files, such as web pages or other HTML-based documents.

• Extracted - Extracted text from native files, such as Word documents, email messages, PowerPoint slides, or Excel spreadsheets.

• ORC/Loaded - Extracted text from image files using Optical Character Recognition (OCR) or loaded text accompanying the images.

• Metadata - Extracted information about the file’s attributes, such as author, creation date, or file format, often used for categorization and searching.

• Transcription - Extracted text from audio or video files, representing spoken words as written content.

• Translation - Extracted translated text from a document, converting content from one language to another while retaining the structure of the original.

• Native View - Extracted representation of a document in its original file format (e.g., Word, PDF, Excel) for viewing without conversion or modification.

• Spreadsheet View - Extracted data from native spreadsheets (e.g., Excel), displayed in a grid format, allowing users to view and interact with rows and columns.

* Bulk Actions Index Button: This Grid button can be used to quickly index native and extracted text from the front end. The record’s Text Set(s) will only be indexed if it has not previously been indexed.

** OCR can be sent to the OCR/Loaded text set or new text sets can be created to store the OCR.

Note

For Processing's "Review Append" exports (that is, native and other material ingested through Reveal Processing and not Review Manager), only the Document_Metadata & OCR loaded will get indexed automatically through the API.

Note

Reindexing metadata: When creating a new custom field and Updating Data to that newly created field (overlaying in RM), Document_Metadata must be reindexed to make the new content visible within the Grid View.

Items not indexed initially (other than Document Metadata, which must be indexed to populate the Grid) may be indexed from within the Reveal application. See Additional Indexing from the Grid below.

Initial Indexing and Index Management

1. Connect to a system running Review Manager and log in as an Administrator.

2. Open the Create pane.

3. Choose Indexes.  

4. In the Project Name field, use the dropdown menu to select your project.

5. In the Index table select the set of loaded data identified in the Import File column to be indexed. You may use Contains or Equals filters in the column headings to refine the list for selection.

6. To alert teams and/or users upon completion of an indexing job,

   1. select Options button and

   2. go to the Notifications tab to choose the appropriate recipients.

       The General tab offers settings to Detect Color, Add Alerts For Unindexed Documents and selection of Html HD Mode to be used (None, All, or PDFs Only). HTML HD Mode allows for a higher definition rendering of the HTML view for documents that aren't indexed as text. This may result in better fidelity to the original native document. This will result in a slower processing time and larger index size.

      Note

      Detect Color must be turned on here to implement color detection during indexing; otherwise this process must be done manually.

7. Choose the Index/Re-Index button located in the middle of the window beneath the Index table.

8. Select the Document Text Sets you wish to index. By default the Native/HTML, Extracted, and OCR/Loaded text sets are present, but additional text sets may have been added to the current project.

   

9. The Text Set choices will determine the order which the sets will be indexed.

   1. OCR/Loaded will be indexed first, if present, then the Extracted Text will be indexed, followed by the Native/HTML view.

      Note

      The OCR/Loaded or Extracted Text Sets (preferably both) must be indexed to enable keyword search. Indexing the Native/HTML Text Set is necessary for document viewer rendering, but does not provide search capabilities.

   2. This is done to get data into the project as quickly as possible to make the documents searchable.

10. As soon as a Text Set completes indexing, the project will become searchable.

11. If Document Metadata is overlaid or updated, that portion alone may be selected and indexed as a text set without having to reindex all content in the dataset. 

12. When indexing is complete the documents may be viewed in the Native / HTML viewer in the Document Review screen and Document Metadata will appear in the Review Grid.

13. To examine the indexing log for the current or previous jobs in the project, click View Log.
     

14. To get information about the health of indexes, click View Health. The report window will provide details about the index, including the text sets indexed and their counts.
     

Note

For performance reasons there is a hard limit of 16MB expanded text size for indexing documents in the Native / HTML text set. While settings in Review Manager may be set to indicate a larger limit, any document exceeding the 16MB limit will not index and an error will appear in the indexing log. We strongly recommend contacting Reveal Support if encountering this limitation.

Note

The native and text file sizes differ from the expanded file sizes. The expanded file size is the size of the text set created.

Specific documents can be targeted using the Doc List field to specify a List file containing the identified document numbering field (BEGDOC by default). Users can choose between Unindexed and Errors, Non-Error, and All Documents.

Users can choose to change the priority of a specific indexing jobs. The priority of a job is relative, so if a user chooses to make all indexing jobs High priority, the net effect is that all jobs are the same priority and, therefore, there is no high priority.

Progress

Warnings