Rapid Pilot Project Creation

Document Purpose

The purpose of this document is to provide the instructions to create the "Rapid Pilot Project". This project consists of an Enron subset, and a handful of documents to showcase our Review Accelerators (Translation, Image Labeling, and Transcription) totaling ~40K documents. This allows us to have a project for sandbox or training purposes.

Technical Documentation

To access all technical documentation within the Reveal Discovery Platform, please go to our Online Help Portal.

High Level Process Overview

Below provides the high-level illustration of this entire process.

Access to Load Machine

All Load Machine access is administered through Perimeter 81. To access a Load Machine, you can follow the high-level steps below.

1. https://landing.revealdata.com

2. Use your Reveal Universal ID to login

3. Find/click on the icon for your Load Machine by searching for the MSA.

4. If you cannot find or access your Load Machine, please email [email protected] and ask for access.

Note

Please take note of the MSA or 8-digit number in the name of the machine. This is the MSA number for the pilot.

Stage Test Data

There are two steps to stage the test data.

1. Download the test data to the Load Machine.

2. Copy/Unzip the data to the root of the Z:\ drive on the Load Machine.

Download Test Data

Follow the steps below to download the Test Data.

1. On the Pilot’s Load Machine, use the following link in Chrome to download the Test Data,

   1. https://revealdata-public-test-data.s3.us-east-1.amazonaws.com/_RapidPilot.zip

Copy/Unzip the Data

1. After downloading the zip file, copy it to the Z:\ drive of the Load Machine.

2. Unzip this file to the root of the Z:\ drive so it looks like the below.

   1.  

   2.  

Note

It is possible to receive a permissions error when trying to copy the zip file to the root of the Z:\ drive. If this happens, create a folder named '_RapidPilot' at the root of the Z:\ drive, copy the zip file into this folder, and extract the contents of the zip file into this folder. Do not repeat the '_RapidPilot' folder underneath itself.

This data must be on the root of the Z:\ drive as illustrated above. If the data is not staged in this manner, the importing of the natives/text will fail. If other files and folders exist on the Z:\ drive that is fine as well.

Project Creation

Reveal Review Manager is used to administer your instance of Reveal Review. In Review Manager, you can do any number of functions associated with the administration of Review such as create databases, manage users, load large amounts of data, set analytics and more. To see more information about this, please see Reveal Review Manager Overview.

Create Company

The Reveal Company may already exist. If it does exist, please skip this step, otherwise create the Company Reveal using the steps below. If it does not exist, or you want to create a new Company, follow the steps below.

A Company is the highest level of organization in our permission structure. Please create a Company named Reveal.

1. Open Review Manager on the Load Machine

2. Expand Instance Setup

3. Click Companies

4. Click the Add New tab

5. Enter Reveal (or preferred name) as the Company name, and click Insert

   1.  

Create Client

The Reveal Client may already exist. If it does exist, please skip this step, otherwise create the Client Reveal using the steps below. If it does not exist, or you want to create a new Client, follow the steps below.

A Client is only associated with one Company. A Client can have multiple projects associated with it and is a way to keep a logical structure to the projects.

1. Open Review Manager on the Load Machine

2. Expand Instance Setup

3. Click Clients

4. Click the Add New tab, enter Reveal (or preferred name) as the Client name, enter a Client Number (typically a number, recommended to use date created in YYYYMMDD format), associate the Client with a Company, and Click Insert

   1.  

Create Rapid Pilot (RP) Enron Project

After creating or using a previously created Company and Client, follow the steps below to create a new Rapid Pilot.

1. Open Review Manager on the Load Machine

2. Expand Instance Setup, select Projects

3. Click the New tab

4. Enter the Project Name (use ‘RP_’ as the prefix, the client's MSA_Enron as the project name, i.e. RP_81010001_Enron), enter a Project ID (typically a number, recommended to use date created in YYYYMMDD format), associate the Company/Client, choose the Reveal InControl Standard Template, and UTC for Time Zone.

5. Click Create Case

   1.  

Note

You will always choose Reveal as the client and company, but the MSA in the case you create will be different. The MSA provided in this document of 135176 is there for instructional purposes.

User Setup

After creating the project, there are a few things that must be done prior to importing the data.

Create Review Account and Add to Rapid Pilot Project

Create a Review account for yourself and make your user a Company Admin as detailed below.

1. Open Review Manager on the Load Machine

2. Expand Project Setup

3. Click Users

4. Click Add

5. Click Create New, fill in the Create a new User ID form, and click Create

   1.  

6. Choose the Review Account, and click Add

   1.  

7. Change Group to Administrators

   1.  

Company Admin Rights

A user must have Company Admin rights to create users or a project within Review. To make a user a Company Admin follow the steps below.

1. Open Review Manager on the Load Machine

2. Expand Instance Setup, and select Companies

3. Click the Users By Company tab

4. Click Add User, and choose your user

5. After adding the user select the checkbox Is Admin.

   1.  

Checking for Company Admin Rights in Review

If a user does not have access to the Load Machine, but wants to check for Company Admin rights, the user can check within Review.

1. Log into Review

   1. You can get the URL to Review to any environment in Review Manager by expanding Advanced Options, click System Settings, and in the Value column type 'indexserverserviceclient'. When copying this URL result, exclude '/review_indexserver' from the result.

2. Click Admin

   1.  

3. Click the Menu button in the upper left-hand corner and look for Company Admin.

   1.  

Note

If Company Admin is not visible, a user with access to the Load Machine will need to grant Company Admin rights via the steps above.

Import Data

We will import data into the project. This requires importing the DAT file into the system and creating indexes. To do this, follow the steps below.

Note

The project in the screen shot is for illustration purposes only, and you should be choosing your project you created.

Field Creation

To load the data set change the below settings.

1. Open Review Manager on the Load Machine

2. Expand Project Setup

3. Click Fields

4. Change DOMAINS from 4000 to 0 which is the max characters allowed for a field.

   1.  

Import Mappings

A new field mapping will also need to be created to load the data.

1. Open Review Manager on the Load Machine

2. Expand Project Setup

3. Click Import Mappings

4. Browse to the following DAT file

   1. Z:\_RapidPilot\PilotDataProcessingFolder\Exports\0001_EXPORT01\EXPORT01_loadfile.dat

5. Click Match All Fields  

   1.  

6. Sort by the field Table Name, click in one of the fields that say -- Select -- and find/remap only the fields that show here as the did not map properly (there will not be that many). When this is complete, there will be no more fields remaining in the drop down.  

   1.  

7. After mapping the unmapped fields, save this mapping as the Rapid Pilot Mapping

   1.  

Note

All fields will not map. You will only map the fields that show via step 6 above.

Documents Import

To import the DAT file, follow the steps below.

1. Open Review Manager on the Load Machine

2. Expand Import

3. Click Documents

4. Choose the Rapid Pilot project

5. Choose the Rapid Pilot Mapping

6. Import 1

   1. Import Data File: Z:\_RapidPilot\PilotDataProcessingFolder\Exports\0001_EXPORT01\EXPORT01_loadfile.dat

   2. Click Import Data

   3. This will load 40,656 files.

      1.  

      2.  

      3. 

         1. You can ignore this error and click OK.

7. Import 2

   1. Repeat the same steps from Import 1

   2. Use Import Data File: Z:\_RapidPilot\PilotDataProcessingFolder\Exports\0002_EXPORT02\EXPORT02_loadfile.dat

   3. This will load 86 files.

Note

As shown above, it is possible to receive an Invalid column data error when importing. It is ok to click OK and disregard this error.

Documents Overlay

When this data set was created, there were a few fields that were not within the DAT file that are essential for visual analytics and AI. To get these fields populated, we will need to do an overlay or Update Data for the project in Review Manager. To overlay these fields, follow the steps below.

1. Open Review Manager on the Load Machine

2. Expand Import

3. Click Documents

4. Choose the Rapid Pilot project

5. Choose Update Data

6. Choose CSV

7. Overlay

   1. Import Data File: Z:\_RapidPilot\PilotDataProcessingFolder\Exports\0003_BRS_OVERLAY\BRS_OVERLAY.CSV

      1. Please note that this is a CSV file and not a DAT file.

   2. Click Update Data

   3. Click Match All

   4. We are only overlaying 3 fields. You will need to find the following two fields and deselect them from the overlay, as Match All will return 6 fields.

      1. Find the field SENT_DATE and deselect it (remove the checkbox from Import for the row) as we should be overlaying the field SENT_DATETIME and not SENT_DATE

      2. Find the field SUBJECT and deselect it (remove the checkbox from Import for the row) as we should be overlaying the field SUBJECT_OTHER and not SUBJECT

   5. Click OK

   6. This will update all files (40,742)

      1.  

Note

As shown above, it is possible to receive an Invalid column data error when importing. It is ok to click OK and disregard this error.

Create Indexes

After importing the data, you need to index the data. Please only do this after Store Pending = 0 as seen below. As part of this process, we grab the native and text files from the load file, and move them to S3 in AWS, or the “store”. Also note there is 1 file that is a Store Error. This was a virus that was removed by AV, so this is expected. This process will take some time, so this may be a good time to take a break for about an hour.

Please note the project in the screen shot is for illustration purposes only, and you should be choosing your project you created.

1. Open Review Manager on the Load Machine

2. Expand Create

3. Click Indexes

4. Choose the Rapid Pilot project

5. Select both indexes

6. Click Index/Re-Index

7. Click Native / HTML, Extracted, OCR / Loaded, Document_Metadata, and click OK

   1.  

Create Document Folders

After kicking off indexing, run the following process in the Create section of Review Manager.

1. Document Folders

   1. Choose the Rapid Pilot project

   2. Select both data sets

   3. Change Document Folder Field to RELATIVEPATHPARENT

   4. Click Run

      1.  

Monitor Progress

1. You can monitor the indexing progress in the Review Manager, or you can go to Menu -> Jobs -> Index, select View All Jobs, and you will see the indexing process of the import. Make sure the counts in your project match the counts in the screen shot below except for Total Errors.
   

   1. 

2. Run the term depart* to make sure 5,260 documents are returned.

   1.  

3. Make sure the data loaded properly into AI by going to Menu -> Jobs -> AI Document Sync. If this shows complete

   1.  

4. Everything will be complete when the Candy Bar displays counts on the Dashboard, and that the index Brainspace connector overlay completed as well.  

   1.  

   2.  

Note

It can take upwards of 20 minutes for the Candy Bar and/or the Brainspace connector overlay to display after the AI Document Sync has completed.

Run Final Processes

There are 3 processes to run if you need to showcase the Review Accelerators (i.e., Image Label, Translation, and Transcription). Please note the project in the screen shots is for illustration purposes only, and you should be running these processes in the project you created. When the Label, Translation, and Transcription jobs complete, you will receive an email for each job.

Field Profile Management

By default, field profiles are not assigned to roles out of the box. This makes the Default Field Profile inaccessible for all users. To fix this, follow the below steps, and you can use the Default field profile.

1. Login to your project via your browser in Review

2. Select Admin

3. Click Fields -> select Default -> click Edit

   1.  

4. Select Original Administrators -> click UPDATE

   1.  

5. Switch to Grid, and choose Default if not selected

   1.  

Label

Run Image Label on the following images for demo purposes.

1. Login to your project via your browser in Review

2. Select the Documents Folder Documents -> Dropbox -> Group Share -> Images, and this should return 49 files

   Note

   If the top level folder does not return 49 files, click the dropdown arrow and initiate the process outlined below on each sub-folder.  

3. Click the Label button

4. Click LABEL

5. You can monitor the progress of this job in the Jobs section (click Menu and choose Jobs). When the job is complete there will be a green check mark.

   1.  

Translate

There are two sets of documents for translation purposes of Japanese and Korean. Below illustrates how to translate the Japanese files. The process should be repeated for the Korean files as well.

1. Login to your project via your browser in Review.

2. Select the Documents Folder Documents -> Dropbox -> Group Share -> Foreign Languages -> Japanese, and this should return 30 files  

3. Click the Translate button

4. Next to Destination Text Set click the + Create new button, enter the following items, and click Submit.

   1.  

5. Back on the Translate Documents settings, choose the following options.

   1.  

   2. If the Destination Text Set that was just created does not show up in the list of text sets to choose from, logout/login, and the text set will be available.

6. Repeat this process for the Korean files (Documents Folder Documents -> Dropbox -> Group Share -> Foreign Languages -> Korean), but translate from Korean to English. Please add the Korean files to the same Text Set of Translation, and do not create a new Text Set.

7. You can monitor the progress of this job in the Jobs section (click Menu and choose Jobs). When the job is complete there will be a green check mark.

Transcribe

There is 1 deposition video to transcribe.

1. Login to your project via your browser in Review.

2. Select the Documents Folder Documents -> Dropbox -> Group Share -> Videos, and this should return 1 file

3. Click the Transcribe, and click TRANSCRIBE (leaving all default settings)

   1.  

4. You can monitor the progress of this job in the Jobs section (click Menu and choose Jobs). When the job is complete there will be a green check mark.

Show Transcription

After the deposition video completes, turn on the transcription to show it during a demo.

1. Login to your project via your browser in Review.

2. Select the Documents Folder Documents -> Dropbox -> Group Share -> Videos, and this should return 1 file

3. You can monitor the progress of this job in the Jobs section (click Menu and choose Jobs). When the job is complete there will be a green check mark.

4. Launch this file in Review by clicking on the row.

5. Click the CC Show Transcript button  , and this will show the text next to the video when it is played. This setting will stick for all future demos of this file.

Setting up Rapid Pilot for Demonstration Purposes

While the Review Enhancers are executing, several additional tasks can be done to get the project ready for demo purposes which are outlined below.

Add AI Model

Apply an AI Model from the Model Library for demo purposes.

1. Login to your project via your browser in Review

2. Click into Project Admin -> Tags

3. Click Add Tags and Choices

4. Enter and select the following information for the tag:

    

5. Return to your main Project screen -> Click Supervised Learning

6. Click the gear icon by the AI Tag created (Junk)

7. Scroll through the AI Library Models and add the "Out of Office" AI Model

8. Click Run Full Process -> Click Save at bottom 

9. Click the Review Icon on left panel, return to dashboard.

Note

The project's documents will now be scored with this AI model as part of the current Classifier which will take some time to run the process before it displays on the dashboard.

You can replicate these steps to create another AI Model for demo purposes or even combine two similar models from the library to "pack and stack" into one uber AI Model to further leverage the AI Model Library.

For more information on applying or leveraging AI Models, please see or help documentation or blog posts.

Creating Work Folder for Key Documents

For demo purposes, start selecting documents from Rapid Pilot Dataset to highlight key differentiators and place into Work Folder.

1. Login to your project via your browser in Review

2. Next to Work Folders on top left, Click the ellipsis

    

3. Click Add Folder -> Name Folder "Key Docs" -> Click to Grid View

4. Click into Search Toolbar at top right, Click the ellipsis

5. Click Add Condition -> Type Begin Number -> Click Begin Number metadata field

6. Copy and paste the following Begin Numbers into the pop out window:  

   1. DEMO-037138

      DEMO-040676

      DEMO-040720

      DEMO-040742

7. Click Add to Search -> Click Search -> Results should look like documents below:

   

8. Click Update -> Action to Take -> Check Folders -> Click dropdown to Work Folders -> Check off Key Docs -> Click Submit -> Confirm

   1.  

Note

The work folder and documents selected can be selected based on preference for demo purposes - just a suggest set of documents to highlight Review Enhancers.