Processing - 10.1 Release Notes

Updated look, similar feel – Reveal Processing 10.1 continues to sport a Black skin option for both the Manager as well as the Agents. The new release number dovetails with enhanced integration with the Reveal Review platform.

A. Feature Additions:

• Ability to create separate Instances within the same SQL environment. This allows for greater control and system management internally to segregate projects. Each instance will contain a Manager Database (identical to the Tunnelvision database) and the instances themselves can be named as seen fit. All databases created in these instances will have unique identifiers to link them only to their specified instances. Users will have the ability to change between instances in the Home ribbon of the Discovery Manager.

• Reveal Processing Agents now can run as a service making them easier to manage and control.

  • Discovery Agents have UI level access to input Service account information as well as to start/stop the service on the Discovery Agent machine.

• Job Recovery – The addition of this feature allows users to more easily resume Distributed jobs that were paused or interrupted and allow the software to resume these jobs as opposed to requiring a direct interaction in SQL. This feature will be automatically run on any resumed job to locate any tasks orphaned due to the pause or interruption. A button in the Environment Module can be used to recover or resume errored jobs to automate the tasks.

• Environment

  • Import Jobs – The addition of both the ability to refresh an Archive Explorer as well as the Error Explorer adds better understanding of current processes running. The Archive Explorer will afford users monitoring jobs in the Environment module the capability to monitor the process of individual archives that are decompressing. The Error Explorer will capture the associated error messages for an import (which do not indicate job failure) but will not be auto-populated which was shown to slow down processes in SQL when viewing.

  • Update Projects Tool — This tool added in v2019 is now more powerful as it has been expanded to allow for 5 simultaneous project level updates at a time. This eliminates the need to update projects by opening each in the UI one at a time. While the single level approach will still work, the Updater tool will allow for better scheduling and management of the update process.

  • Project Migration Stored Procedure — A SQL stored procedure named u_MigrateProjectPaths has been added to every project that will automate all SQL pathing changes needed for a project when the Processing folder is moved or renamed. This eliminates the need to run multiple SQL scripts to achieve this result.

• Projects

  • Ability to automate SCANPST on data ingested -- This feature will attempt to run SCANPST on any PSTs added to an import up to 12 times if continual errors are found. If no errors are found when running the processes will cease. This feature adds in much value in eliminating many errored PSTs on import. The feature can only be run when the following version of Outlook (or later) is installed as Microsoft will not support it in previous versions.

    • Supported version is Outlook 2016 (16.0.12527.20880 or later).

  • Ability to enable/disable the Term derivatives being added to the project -- The addition of Term derivatives can slow down the Import and Import QC processes excessively. The derivatives are added visually in the search module and add do not affect wildcard searching in any way.

  • dtSearch Batch Size has been re-added into the project creation form. This gives users the ability to lower or raise the batch sizes given. This can be very helpful for larger text sets that are added.

  • Update Visual Counts Automatically – This option when selected will not automatically update the visual counts of term hits in selective sets. This is only a visual feature as the hits will still occur. Having this option selected will speed up the Import and Import QC processes. The option can be toggled on and off as needed.

• Import

  • Enhanced PST processing -- The underlying methodology for PST processing has been sped up anywhere from 3-7 times faster.

  • Multi Custodian assignment — This allows users to not only create Custodians on the fly, but to also select multiple Custodians to pre-assign data to for any given import.

• Import QC

  • Both Text and Metadata are no longer generated by default when launching Import QC. The addition of a button to generate these is now in place. This is designed to allow quicker access to any given set of data being QC’d.

• Imaging

  • Removed local path information seen when manually printing files via Internet Explorer.

  • Added the ability to exclude File Recognition Errors in any scope chosen in Imaging Jobs.

• Export

  • Dedupe Scopes — This new feature allows users to define their own custom level of deduplication to apply to any given scope. The custom scopes created can be used in exports as well as reporting. These scopes can also allow previously exported data to be added to create new master level hierarchy inside the scope itself.

B. Bug Corrections/Enhancements:

• Environment

• Additional Indexes and optimizations have been put in place to allow for easier navigation inside the module when monitoring jobs.

• Issues encountered where config files are stored locally on machines now have additional security measures in place to disallow any unwanted changes due to network changes or interruptions.

• Better measures in place to clear the temp location when re-starting the applications.

• Projects

  • Archive/Restore Functionality working as expected.

  • Additional checks in place for projects with the Save Native option enabled that will purge the source PST files after successful ingestion.

• Imports

  • In Embedded Object projects, the text for the embedded objects will be found only in the stand-alone embedded file’s text and no longer in the parent text as well.

  • Improved performance for non-email archives, specifically .7z, .rar, and .tar file types.

  • Corrected issues with invalid dates pulled from Internal metadata for Efiles.

  • Corrected the inability to use network temp locations for Forensic Image processing.

  • Improved processing reliability for multi-part L01 Forensic Images.

  • Corrected issues where pausing and then resuming imports would cause duplicative data to be added to the import.

  • Language ID will be run on files that have completed the OCR process.

  • Better suppression of .emz and .wmz files listed as archive exceptions.

  • Update the Tunnelvision..FileTypes table to match current supported file types as recognized by the file recognition process and the third party app used in the file recognition process.

  • Additional measures in place to disallow files to be added as “source files” that have no content.

  • Corrected issues where some .eml files were being recognized as MBOX files.

  • EfileMetaData uniformly in place so that the dates pulled are region agnostic and will be similar in any region with different settings.

  • In Imports where OCR is to be run during Import, at completion of the import there will not be any files listed as OCR Pending.

  • Over-use of the OnBehalfOf field corrected so that the SENT field for emails will not improperly be set.

  • Enhancements added to attempt better parsing of valid email participants in x400/x500 addresses to match more in line with the standard SMTP email’s addresses. Normalization processes for the addresses use multiple paths for the end result.

  • All files successfully OCR'd will be routed for Language Identification as expected.

• Import QC

  • Several Indexes and enhancements have been added to make opening and navigation in Import QC quicker.

  • Import QC will no longer be available on an Import that is In Progress.

  • Pausing and then Resuming Import QC Jobs will no longer cause stalls at the end of these jobs.

  • Because of the enhancements added, Import QC jobs that have files that have extremely large text files will no longer cause long delays in opening QC.

• Search

  • Additional checks in place to ensure that there are no files that reference or belong to Indexes that have been removed. These files will instead appear as needing to be indexed as expected.

  • Fielded Searches working as expected.

  • Corrected an issue where the file’s MD5 Hash was being indexed.

• Selective Set

  • Added the Index Status of Decrypted under the File PropertiesàIndexStatus tab. This will add in any non-archive files in which a password was supplied that properly decrypted the file to allow for processing.

  • The use of Export File Id in File ID Lists running as expected.

• Imaging

  • PDF Comments will be seen in successful images when selecting the option to print the PDF comments. They will be seen in the final image(s).

  • Corrected issues where Manual imaging would stall indefinitely.

  • Emails can be manually printed via Outlook. Previously these were handled by printing the emails through Word.

  • Issues in Image QC where duplicates in the scope of the Imaging job caused visual issues in the grid have now been corrected.

  • Enhancements on PowerPoint Imaging were performed using data that eventually would crash the Agent running the job.

  • For PowerPoint files imaged using the DOJ/SEC option, no longer see speaker notes overwriting the slide; they are properly sized in the image.

• Reports

  • Corrected an issue in the Snapshot report where specific chosen options selected together caused the report to produce a fatal error.

  • Corrected visual issues in the Duplicate Report in relation to the Index Statuses of the files.

• Export

  • Corrected visual count issues in relation to Imaging jobs.

  • A single apostrophe in an export name or export location will no longer cause export failures.

  • Enhancements and indexes have been put in place to speed up the processes for Initialization of exports in larger projects. Projects with large export set tables are taken into account.

    • Foldering structure within the export sets table enhanced via updates to different stored procedures and code.

    • Expansion of family members that come from family level selective sets that previously caused degradation has been corrected.

  • Changes have been made to ensure that the UNREAD status in the EMAILREADSTATUS field is properly written.

  • The export base path is properly applied when using the default instance path on project creation.

  • Corrected an issue where the fields DUPEPATHS and DUPEPATHSWITHORIG were not populating as expected.

  • Export failures where files in the export have a file type = 0 have been resolved in the import process, but if ever present will no longer cause stalls/failures in the export process.