Processing - 10.4.1 Release Notes

Fix

Assignment

Custodian: Assign files - Right click not working on Continuous Content section

When right clicking a parent section of the Custodian file tree, an option to assign all documents under the section should pop up, this was not the case when right clicking Continuous Content section.

Fix

Authorization

Processing Review Account Info Allowing Invalid API URL

In an environment linked to Review if a user uses an API URL that starts http instead of https, no error or warning is thrown. If the user tries to export to Review, they get an error that doesn’t point in the correct direction. We now throw an error when the user applies the invalid API URL that informs the user to launch the Review account button in the home ribbon and change the API URL from https to http.

Fix

Authorization

Prompt to Enter Credentials or Pop-Up Error When Service Is Started Without Credentials

Added pop up error to Discovery Agent to prevent agent from starting if user has not signed into the program, preventing the program from crashing.

Fix

Deduplication

Deduplication of Files When Building Dedupe Scope from Previous Exports Is Incomplete

Adding previous sources does not work to seed the dedupe scopes retroactively. You can only use these scopes if they are seeded by exports where the scope is used as the deduplication method. To work around this issue, currently the client can manually seed previous exports into the dedupe scope in a placeholder export.

Fix

Environment

Pass NIST database version to Environment Settings

Added code to pass NIST database version into business layer code and DAL repository call to create environment database and save environment settings into EnvironmentSettings table in instance environment manager database.

Fix

Environment

Processing: Environment - Success Imaging Job graph shows results as pending

After successfully completing an Imaging Job, the Graph that shows the behavior under Environment > Success menu remained yellow, indicating that all the files were still pending. Now identifies when the job is a PDF imaging job to update the chart based on PDF status instead of Imaging status in JobProgressImaging.

Fix

Export

Apostrophe in Custodian Name Causes Issues on Export

Problematic characters in custodian names no longer cause issues on export folder creation.

Fix

Export

Export Error (Parent\Child)

An error message caused a client’s export to fail. There was an associated FileID and the belief was that a Text file failed to copy to the export location from the Processing location. The error should be skipped past the offending file and the export should continue and then leave error message at end of completed export.

Fix

Export

Export: Run Export - Not showing estimated export size

Estimated export size not showing up in the Export Summary window. This seemed to happen only for RSMF files and/or when files size was equals or greater than 2GB.

Fix

Export

IMAGE Export BEGBATES Numbering Issue

When a user exports out images and uses the BEGBATES option for numbering using v10.3.1.247, the export completes but only one text file or native actually exports even though the export snapshot indicates all files exported. The Images export without issue. All natives and text should be at export location and properly labeled (numbered).

Fix

Export

Processing 10.3 allows you to export (Review Append) the same document twice to the same Review case.

While duplicate documents can be exported to Review, the user should be notified with the warning which already exists in code, and the export should not be added to the 'Exports In Review' population.

Fix

Export

Remove Non-Printable and Control Characters from all Efile Metadata

Users are able to export load files to Review, with no issues with non-printable and control characters.

Fix

Export

Reveal Discovery Manager (on-prem): issue in pulling of documents

Pulling documents from Reveal Discovery Manager using ExportFileID or ExportBegBates, user tried to export the specific Exported Volume as an overlay and also got a Zero Document result. Found that when adding a fileid list to a Selective Set, the new table created in SQL altered the string in the 'FileID' column that is used to pull the docs. Remediated by taking note of the name of the Selective Set and the clause of the Selective Set that has the FileID list, obtaining the database name from the About button on the ribbon, then opening SQL to set focus to the affected project.

Fix

Export

Trailing Space in PST Folder Path leads to Native Export Failures

When a trailing space exists at the end of a folder path within a PST's folder structure and documents in said folder are exported out to Natives Only MSG, the natives will not copy. Ultimately a trailing whitespace is not allowed in the folder path and folder creation breaks down for the export at that point. Remediation for native MSGs generated with the folder structure of the PST: Any invalid folder path characters are addressed at export time in a manner that allows the documents to go out.

Fix

Imaging

Imaging Jobs - The process status graph does not show the correct status

When performing an Imaging job, in the Environment Module if the job was selected the status graph was not reflecting the actual status even when the job finished.

Fix

Import

Import: New Import: Import from S3 crashing

Fixed issue when user selects S3 bucket root as import data path.

Fix

Import

Long path issues while processing the data

When trying to import the forensic images, the Discovery Manager was unable to access the files in their location due to the length of the path. Because of this, the files were unavailable for import and would show an error; no additional work could be done on the file. Users can now ingest data from forensic images that contain a long file path.

Fix

Import

S3 Browser Button not disabled after import and misc.

Import from File Explorer Button is now disabled when Import is selected.Tables in Create OCR Job not editable. Preview Window bottom panels are visible on open and do not show Close button. Selective Sets table is visible on open.

Fix

Import

S3 Pathing Captured for Upload Jobs 10.3

After applying v10.3 Processing, all uploader jobs capture Import Paths pointing at S3 locations as opposed to the typical paths. This carries over to many fields sent to Review. Updated to replace manual workaround.

Fix

Import QC

Dock Import QC Selection

If a client opens Import QC for Import ID = 115 (as example), when the QC Window is closed the selection for Import Icons returns to the very first Import every time. The selection choice now remains at the Import Icon last selected.

Fix

Import QC

Missing Native File Remediation Does Not Work

Forcing ImportStatus=2 on a missing file via SQL (or running Validate Natives) on a case where files are missing from the project folder forces it to appear as QC Flag Category ‘Attachments’ or ‘Missing Native’ from Import QC. Reprocessing these files does not work to restore the native even when the source native is available in the original stored location. Until we can remediate the underlying cause for Non-AV Quarantine Missing Files from S3 - need a path to restore backend files to a case.

Fix

Import QC

Not Capturing Stop Time for Majority of QC Jobs

Not capturing a Stop Time for majority of Import QC Jobs. The ones actually captured have values in the Processing Jobs table in the ProcessedDocs Column. Need to make sure that we capture a Stop Time for all QC Jobs.

Fix

Import QC

Replacing Files Individually Via Import QC

Replacing files via the Import QC ‘Add to Batch’ functionality hung indefinitely and did not work. Clients are now able to use the file replacement functionality in the software for best practice critical workflow path.

Fix

Import QC

System Clock Not Adjusting Back to Default After Import QC Job

In Import QC, clock was not reverting back to the machine timezone once the import completed.

Fix

Projects

Discovery Manager: Project Creation - Not able to Create a new Project running DM for the first time

Addressing an issue where after a fresh installation of Discovery Manager attempts to create a Project cause an error.

Fix

Projects

New Project Creation form: The Template & Client dropdowns contain duplicate values

When the user tried to create a new project in the Creation form when the -Default Settings- option was selected in the Template dropdown it caused the values listed to be duplicated in the Client dropdown and in itself. Fixed template drop down box that was duplicating the same instance in the template menu as well as in the client drop down box.

Fix

Reports

Spelling error in the message of the Reports module button

Fixed misspelled text for Generate Interactive Report button on Report menu.

Fix

Selective Set

Image Begbates Fatal Error In Processing when Adding to Selective Set

The below fatal error appears when adding BEGBATES numbers to Selective Set with FileID functionality: --------------------------- Fatal Error --------------------------- Application Error One or more errors occurred. [Execute] SQL Error: Unclosed quotation mark after the character string … FileID List Add to Selective Set is now successful for Image Bates type, and if the values do not exist or values aren't found, we create an empty or partial Selective Set to handle gracefully.

Fix

Selective Set

Large Lists of Domains/Participants Cause Fatal Errors in Selective Set

Resolved fatal error stemming from extremely long query of elements on Selective Set clauses.

Fix

System

Home: New Project - Error with Review Credentials

Addressing issue where Review credentials error with creating a new project after connecting to Review for the first time.

Fix

System

Instance Settings, General Settings: Folder Browser not working

A folder browser dialog is presented exactly like the folder browser from the Imports Module.

Fix

System

Issues Creating New Instance

Currently, new instances cannot be created through the UI. When evaluating in 10.3.1 and 10.3.2, the behavior is not consistent. 10.3.2 will display a spinning progress bar whereas 10.3.1 will throw an object reference error message. This issue needs to be fixed in 10.3.1 after which the fixes can be merged into 10.3.2 and that branch revaluated. Acceptance criteria: 1.) A new instance can be successfully created when installing to a new environment. 2.) A new instance can be successfully added to an existing environment. 3.) A user should be able to create a new instance, exit out without selecting that new instance, and remain in the prior instance.

Fix

System

Japanese Custodian Characters not Properly Seen in TextFiles.dat or Text Load File Fields

Within the pathing to the text files, the custodian name should appear as LASTNAME, FIRSTNAME correctly with the Japanese characters. Additionally, these characters should be correctly represented in the TEXTFILE and RELATIVETEXTFILE export load file fields and the generated cross reference TextFiles.DAT at the base of the export.

Fix

System

Non-Unicode Issue Collection Location

Client reported an issue when adding Japanese characters into a Collection Location for Custodian assignment where multiple spots in the database do not interpret the values accurately, propagating into the exported load file options. This has been fixed.

Fix

System

On-Prem Review Integration Broken

On premise Review integrations where Processing is integrated with an on-premise Review installation were not functioning as expected. Two issues were identified: 1.) Review append exports were failing. 2.) Valid credentials were returning validation errors in the Discovery Manager after a period of time. The issue was that the Review auth token validity timespan has been reduced. Historically, when Review credentials were used without an SSO integration, the auth token was associated with the Processing job and used to complete work. With the reduced timespan, that auth token was invalid by the time the agents interacted with Review. Similarly, on the Discovery Manager the auth token acquired when opening up the application was expiring and not refreshing correctly. In order to address this, the refresh mechanism in a non-SSO environment has been corrected. Additionally, the username and (encrypted) password are now export job settings which are utilized by the agent to authenticate against the review API.

New

Import

Deprecation of the Mounting of Forensic Images

Certain Forensic Image Types are processed through an approach where they are mounted as a virtual drive, and then the contents of that drive iterated through. This approach is also utilizing outdated third-party components that are error prone. This approach should be refactored such that forensic image processing is more stable, and achieved without mounting a drive within the operating system.

New

Projects

Discovery Manager: New Project - Error Loading Field Mappings

Right after creating a new project linked to Review a new error dialog pops up indicating that there was an error with the field mapping.

New

System

Discovery Manager throws Exception when in clean environment

If the Discovery Manager was previously installed, remove all connection string traces from the registry and appdata folder. Install the Discovery Manager and connect to a local SQL instance with no preexisting databases.

New

System

Retaining Installs Folder

When using the update folder to update a machine's version, we keep a copy of the Windows installer file at the C:\ProgramData\Mindseye\Installs location. While this only will really affect us as we update quite a bit, this could affect clients for local space if they were to use this folder repetitively. Should only keep a copy of what is necessary.

New

Templates

Global Preview Templates do not Work

In v10.3.1.247 if you launch either Import QC or Preview via Selective Sets or searches and create a template at Project level, it visually creates both a Project level template and a Global template, but the Global one is not seen in other projects.

Update

Deduplication

Project Settings, General Settings: Calculate Custodian Dedupe

A project level setting to ‘Calculate Custodian Dedupe’ has been added which when not selected during the custodian assignment process will not calculate custodian duplicates, resulting in a significantly faster assignment time. Custodian level duplicates are otherwise no longer calculated for every document when assigning data to a custodian or unassigning data from a custodian unless found necessary in order to support custodian level dedupe. When selected, custodian dedupe will not be an option for exports or reports. If this setting is turned on mid-project, all CustodianDupeIds will be calculated across the project.

Update

Environment

Client Name in Environment Module not seen in Default View

When looking at jobs in the Environment Module (Processing Jobs tab) in v10.3, the Client Name was not in the default grid view. Now returned to the default view.

Update

Environment

Environment Update Dialog

Instance Prefix and Manager Database name are presented clearly on the Update Confirmation Dialog. Exiting out of the Update Confirm dialog without updating will allow for the selection of a different instance. A server dialog will pop up if DialogResult is not OK.

Update

Environment

Remember Selection in Environment Module

When clicking refresh in the Environment Module, code was added to temporarily store the selected processing job and processing machine; after refreshing the information, the previous selection is reapplied.

Update

Export

Adapt and Expand Export Fields for RSMF

Current email export fields around participants will need to be extended to account for participants within an RSMF conversation where possible. In addition, we will want to add the following new export fields: - CONVERSATION - This should be populated with the display field in RSMFConversations for all exported events. - CONVERSATIONID - This should be populated with the Id field in RSMFConversations for all exported events. - CONVERSATIONPARTICIPANTS - This should be all participants in a given conversation. - CONVERSATIONSLICESENDERS - This should be all participants who sent messages within a given 24-hour window.

Update

Export

Add Native Export File Resource for Natives Only

Native Only Exports will need to allow for some export capabilities of RSMF data, as this data is being processed and would otherwise be omitted entirely from a Natives Only Format export. A quick and easy answer to this is for parent RSMD Events to create an EML.

Update

Export

Australian Protocol PDF Branding (Bates Stamping)

Export with Australian branding for PDF images allows user to generate PDF images with branding reflecting user selected format of numbering on each page. It includes next types of numbering depending on settings user selected on Export screen Numbering format panel.

Update

Export

Core Australian Protocols Export with PPP.BBBB.FFFF.NNNN

New class NumberingSettings added to store Australian Numbering settings. New member of ExportSettings class of type NumberingSettings introduced.

Update

Export

Export Field: PDFExportPath

When a user is specifically exporting using a Parent/Child export and when exporting a PDF image, the PDF export path will be filled. This field has the full path to the exported PDF file.

Update

Export

Extend Metadata Available for Image Placeholders

The List View Item was replaced with a grid view, now all the Export fields are being populated into the grid.

Update

Export

Overlay Export Fails – Subquery Returned More Than 1 Value

Allow export of files multiple times with overlay creation even if only one BEGDOC can be used based on the 'In Review' population. It is expected that the overlays will only work with one version of the BEGDOC per this feature's intended functionality.

Update

Export

ProcessingToReview ECA

A new method was created to enable the Processing to Review ECA template to be created when it does not exist. With this method we are able to remove the Natives. In addition, the UI was updated with "For No Documents" in the Export Natives selection.

Update

Export

RSMF Rendering for Export Purposes

The current deliverable for RSMF at export time is a PDF rendering and text conversion of a 24-hour window of a scoped conversation. This rendering should include inline images, icons, and avatars as appropriate. Responses and Reactions should be indicated with a tab structure. Given a FileID for a short message within a conversation, create a rendering of a 24-hour window of messages within the conversation. Ultimately what is developed here is what will be utilized at export time.

Update

File Types

Adapt Uploader report to RSMF

Documents are counted as Efile, Email, ShortMessage or attachments, and only one of those categories.

Update

File Types

Add Continuous Content to eFile/eMail Designation

Currently throughout the Discovery Platform, Documents are put in either an eMail bucket or an eFile bucket. This bucket designation is reported back at numerous locations. An additional category, Continuous Content should be added. A ProjectFile should exist in only one of these buckets.

Update

File Types

Ingest RSMF from unstructured data

Improved positive identification of RSMF as a distinct file type.

Update

File Types

Process Multipart 7z Files

Multipart 7z files exist in data processed by Reveal Processing but is currently unsupported. In order to successfully process this data users must go through convoluted processes within Import QC to successfully extract the archive contents. Users would like the system to seamlessly process this data. Added new FileType 10012 for MULTIPART_7Z. Modified i_AddFile to detect FileType 10012 as Archive MultipartFileUtil.cs to get Multipart FileID (also unit tested-Test_Get_Multipart_FileID method). Adaptations when copying files, extraction manager and DistributedImport.ArchiveObject classes to consider new FileType 10012.

Update

File Types

Process Multipart RAR Files

Multipart RAR files exist in data processed by Reveal Processing but is currently unsupported. Users would like the system to seamlessly process this data. Add new FileType 10013 for MULTIPART_RAR and add file type 10013 in i_AddFile sp condition to be Archive = 1. Make adaptations to support Multipart RAR files in MultipartFileUtil.cs, DistributedImport.ArchiveObjectc.s, DistributedImport.CopyFile.cs, DistributedImport.CopyFile.S3.cs, ExtractionManager.cs.

Update

File Types

Process Multipart Zip Files

Multipart ZIP files exist in data processed by Reveal Processing but is currently unsupported. In order to successfully process this data users must go through convoluted processes within Import QC to successfully extract the archive contents. Users would like the system to seamlessly process this data. *FileTypes.SQL -> add line to insert new fileType 10011 for MULTIPART_ZIP. *i_AddFile -> sp change to include filetype 10011 to identify as Archive (update ProjectSchema.sql and in TemplateProject in PSTAG-DB-001 server). *New class in Utils/MultipartZiUtil.cs to handle multipartzip methods.

Update

File Types

Process RSMF.ZIP from RSMF files in a format that Review requires

During Archive Processing, Filetype 10008 (RSMF_ZIP Files) requires special processing from standard zip files. Conversations are created in 24-hour chunks as a single ‘document’ and with the associated metadata are provided as a record in the database. For some clients, conversion to a PDF may be preferred. If a conversation spans multiple 24-hour periods, the relationship will be identified. There will be a header in each PDF that lists some of the key information, such as communicators (participants), time period, communication type, multi-document number tracking.

Update

File Types

Process VHD/VHDX Files

Added new filetypes to handle video extraction.

Update

File Types

Processing Super Type Added into Review

In Processing files are grouped under Super Types such as Word Processing, Spreadsheets, Email, etc (full list below). Processing uses the super types in multiple places to include Import QC to group files together and to map these values to the Application Type field in Review. • DocType 0 - Word Processing • DocType 1 - Databases • DocType 2 - Spreadsheets • DocType 3 - Images • DocType 5 - Videos [may be audio and video combined] • DocType 6 - System Files • DocType 7 - Other • DocType 8 - Archives • DocType 9 - Email • Doc Type 10 – Presentations

Update

Imaging

Create PDFs directly from native files

 In order to support PDF generation for Australian productions, the PDF imaging format has been brought back and validated to allow for conversion of all supported formats to PDF and persist them in the database.

Update

Import

Create Pathing Fields with Subject Email

For PST, OST, and MBOX email, identify when the email has a subject and use it to create the filename within the importpath.

Update

Projects

Remove Review Case Association

A checkbox was added to allow Review Case removal. Once the Update Settings button is clicked, the changes will be applied to the project and next time Project Settings is displayed the Review case will be removed from the project settings.

Update

Reports

Consolidated Billing Report

1.) The report faithfully generates counts for selected projects in an Instance. 2.) The report exports out to Excel in a neat fashion.

Update

Selective Set

Ability to Run Validate Natives on Selective Set

Users are able to run Validate natives set on a Selective Set once it has been defined. Validate natives are run via the Set ID when a Selective Set is made.

Update

Selective Set

Define Selective Sets based upon continuous content attributes

Within Selective Set, a new Tab Page for Continuous Content will be added. The Conversations Tab should display all conversations within the case and include Platform and Type. The filter bar should be enabled so that a user can effectively filter to all conversations from a particular platform (e.g., Teams, Slack, etc). Selecting a conversation (or multiple conversations) should bring all records in from the selection. As conversations are inherently family level, there should be no difference in counts for document vs family level Selective Sets. The Participants Tab seen should include a drop-down for Participant Type containing Sender, Recipient, and All. All should be the default selection. This should be associated with any selected participant in the grid where: • Sender - The selected Participant(s) are senders of short messages within any conversation. • Recipient - The selected Participant(s) is a member of any conversation and could have viewed the message. • All - Effectively the same as Recipient, where the selected Participant(s) is a sender or member of any conversation. A user will be able to search the short message content via the Selective Set Module and will be able to launch a preview of that data. Users will be able to define a Selective Set scope that can be used for export.

Update

System

Ability to Fetch the Passwords Used to Decrypt Files

Gather a listing of the passwords used to decrypt files within Processing: Within the Preview module, an additional field has been added for ‘DecryptionPassword’. This field should populate with a password used to unlock any given file which has been decrypted.

Update

System

Encrypted Password Utility - On Prem

Implementation of a simple utility that, given an unencrypted password passed via command line, the results (to STDOUT) would be the encrypted password that would be usable by both Processing Agents and Discovery Managers in their associated “MDPSettings.xml” file wherever an encrypted password string could be used.

Update

System

Enhancement Request: Record Discover Manager user submitting jobs into Reveal Processing Database

Client request that the logged in OS user be recorded into the Reveal Processing Database. Two new columns in the ProcessingJobs table in the Manager DB suggested: OriginalSubmitter - Text value, populated once, when a Processing Job is originally submitted. MostRecentSubmitter - Text value, processed each time a Project Job is activated (could be different than OriginalSubmitter if Pause/Resume activities have occurred).

Update

System

Export: Custom FileID Sort

Implementation of Custom FileID sort was removed from frmMain. In frmMain, the checkbox to enable Custom FileID sort was moved to Standard Options Tab. A new form (frmCustomFileIdSort) was added, it is displayed after the summary window, only the documents selected to be exported will be presented.

Update

System

Match Review 20-character ProjectID length

Review has been updated to allow 20 Characters. Extended length of Review Project ID to 20 characters in New Project Wizard and Project Settings screen to reflect this change.

Update

System

Programmatically set SSO via CLI tool

Using the DMsetup to programmatically set up SSO via cmd, should take in a baseURL, realm, keycloak, and should encrypt all.

Update

System

Set up AWS S3 Configuration via CLI tool

Able to input S3 information via CLI tool.

Update

System

Update S3 Browsing Experience

Added UI Selection Recursion in frmS3BucketBrowser and added a Size Column. A private method called GetReadableBytes was added to format byte size of files. AllowEdit on both Name and Size Columns are disabled. Also, added a testAuthentication method to show an alert if S3 credentials are incorrect.

Fix

Assignment

Custodian: Assign files - Right click not working on Continuous Content section

When right clicking a parent section of the Custodian file tree, an option to assign all documents under the section should pop up, this was not the case when right clicking Continuous Content section.

Fix

Authorization

Processing Review Account Info Allowing Invalid API URL

In an environment linked to Review if a user uses an API URL that starts http instead of https, no error or warning is thrown. If the user tries to export to Review, they get an error that doesn’t point in the correct direction. We now throw an error when the user applies the invalid API URL that informs the user to launch the Review account button in the home ribbon and change the API URL from https to http.

Fix

Authorization

Prompt to Enter Credentials or Pop-Up Error When Service Is Started Without Credentials

Added pop up error to Discovery Agent to prevent agent from starting if user has not signed into the program, preventing the program from crashing.