Processing - 2024.11 Release Notes

Type

Category

Description

New

Export

Users can now export project-level overview and snapshot statistics from the Processing Overview page. By clicking the export button, users can download a single CSV/XLSX file containing detailed information broken down by each import. This enhancement makes it easier to track and analyze project-level processing data in a convenient format.

New

Import

A Reveal processing user now has the ability to name import jobs similar to export jobs so that they can better keep track of their imports. This is an optional field and not required as it is with export jobs.

New

Production & Export

Users can now export processing job-level overview and snapshot statistics directly from the Processing Jobs page. By clicking the new export button, users will be able to download a single CSV/XLSX file that includes detailed information broken down by each import. This feature provides an easy way to analyze and track processing job stats in a portable format.

New

*Upload Processing

These checks enhance the uploader functionality by introducing three vital validation checks for smoother user experiences during the upload process. Users will now be alerted to any issues with the S3 bucket, write access, or project association before proceeding with their uploads. They may also will help identify and resolve common issues more quickly by reducing silent failures and facilitating easier troubleshooting.

New

*Upload Processing

A new pre-upload analysis step in the DAT file upload workflow provides a summary of records, renderings, and files analyzed before initiating the upload. Users can review counts for extracted text, OCR text, and native paths, ensuring data readiness. Uploads only begin after user confirmation, allowing for early issue identification.

New

*Upload Processing

Introducing the ability to upload delimited files (DAT, CSV, or TXT) for new data loading or data overlays directly via the new web-based Uploader, with or without native renderings. Previously only available through the Review Manager desktop tool, this function is now accessible directly on the web, eliminating the need for a load machine.

New

*Upload Processing

A toggle on the job detail page prevents session timeouts during uploads by resetting the timer with each file progress. Enabled by default, it ensures large jobs over 30 minutes complete without logging out. Disabling the toggle allows standard session timeout behavior.

New

*Upload Processing

The Upload and Process workflow has been implemented., enabling data to be uploaded and processed without automatically promoting it to the review stage. This workflow differs from the standard uploader behavior, where data is promoted back to review after processing.

Update

Field Mapping

Review Promotion jobs now include the "Custodian," "Location," and "Source" fields. These fields are populated based on the values entered during Upload (Web) or Import (Discovery Manager) jobs. The "Custodian" field maps to "Custodian," the "Location" field maps to "Collection Location," and the "Source" field maps to "Collection Source" in Review.

Update

Import

A Reveal Processing user will now be able to export the ‘Import Name’ field populated in the import jobs to allow for better searching and tracking once documents are in Review so that the user can easily identify documents from a particular Import Job.

Update

Project Creation

The option to create a new Review case in Discovery Manager has been removed. In the New Project Creation wizard, users can no longer create new Review cases when selecting “Associate with Review Case.” The Project Settings wizard now only allows associating or removing existing Review cases. All new Review cases must be created through the web UI.

Update

Reports

Generating reports for cases with extensive data has been optimized in this release. For cases with over one million records and more than 1,000 search terms, users will now see a prompt to speed up the report generation process by converting the scope to a FileID List.

Update

Selective Set

The "Beg FileID" and "End FileID" fields will now be shown in the Selective Set and Export modules.

Update

*Upload Processing

To enhance the data upload process, users actively uploading files will now receive real-time progress updates on the current file and a reminder to keep the tab open for uninterrupted uploads. This improvement ensures users have immediate insight into the upload status, minimizing uncertainty and improving workflow efficiency during data ingestion.

Update

*Upload Processing

For web browsers that are not based on Chromium, the "+ UPLOAD" button will be disabled and a message will be shown to inform users that uploading is not available for that browser.

Update

*Upload Processing

The Upload Health Check now includes additional validation for the "Project Link" step. Native file uploads require a linked processing project, while DAT file uploads are permitted even if the health check fails or is incomplete. Messaging for the "Project Link" step has been updated to clarify its impact on file upload types.

Update

*Upload Processing

Improved the layout and functionality of the Uploader Files tab by reorganizing bulk assignment actions, which now appear on the same line as the add folders button. Bulk actions are disabled unless a folder is selected, and a file selection count displays the number of selected files out of the total files added to the upload modal.

Update

Workflows

To enhance the user experience during uploads, the progress bar has been updated. Previously, the overall progress and individual file progress were combined into one visual element. Now, the overall progress will now be displayed using a donut progress bar, with percentage information shown in the center. And individual file progress will now show up to three simultaneous file uploads at once. If more than three files are being uploaded, the progress of completed files will be replaced by the progress of new uploads. Once an upload is complete, no progress will be displayed for that file anymore.

Update

Workflows

The workflow for Upload + Process + Promote to Review has been implemented. When selected, the process uploads the data to processing, processes it, and promotes it back to review, mimicking the current uploader functionality. The workflow is working as expected, as verified by uploading a dataset with the "upload+process+promote to review" feature.

Fix

OCR

An issue was identified in version 92520000 where users selecting "OCR During Import" and "OCR Index Errors" experienced inconsistencies. Although the OCR status showed as "SUCCESS" for all imports, documents still displayed "OCR pending" after initiating searches. This inconsistency originates from the stored procedure, impacting the search process and hindering the client's ability to produce results promptly. Steps will be taken to resolve this stored procedure, and will open a new story for review of all past imports with pending OCR statuses.

Fix

Imaging

Users experience an issue after running a TIFF image job in processing, where the resulting images do not appear in the image preview area, even though the job completes successfully. This happens when navigating to the images tab, and no images are shown despite expectations.

Fix

Document Review

An issue when Review Dataloader jobs is stuck in a "Pending" state, no further exports would complete from the processing side. This results in a bottleneck where subsequent jobs can not be processed, causing delays.

Fix

Document Review

Previously, when using the front-end uploader, the customer selected a Custodian to assign to the data being uploaded. However, after the data was processed and promoted to Review, only a subset of the documents had the custodian assigned. The issue has been resolved, and now the selected Custodian is applied to all documents during the upload process. Once the data is processed and promoted to Review, all documents will correctly reflect the selected custodian without the need for manual updates.

Fix

Document Review

Previously, the bulk assignment feature for Custodian, Location, and Source in the new uploader was not functioning as expected when more than one value existed in the dropdown. Specifically, when the first item in the list was selected (which is highlighted by default), the assignment would not apply to the folders. Now, regardless of the order of values in the dropdown, any selected value can be applied to multiple folders.

Fix

Export

The export settings are incorrectly handling single quotes when being serialized and saved to the database. Specifically, single quotes (e.g., '') in combined fields are being improperly escaped, which results in only one quote being stored instead of two. Now, the issue of escaping single quotes when saving data to ensure consistency with the original input has been resolved. Ensuring that FieldSQL and CombinedFields are handling and persisten data correctly without introducing errors.

Fix

Export

Previously, a null value in the avatar byte array could lead to an exception being thrown. This issue has now been addressed, and null values are properly evaluated and handled as expected, preventing any exceptions from occurring. An additional issue was identified where HTM files were not fully generated before an attempt to convert them to text was made. This led to errors during the conversion process. The issue has now been resolved by introducing additional handling to ensure the HTM files are fully generated before conversion.

Fix

Export

The client and support team have encountered repeated export failures when attempting to generate an overlay of documents for duplicate fields (dupepaths, custiwithdupes, etc.) in both Review(Overlay) and Overlay formats. The export errors suggest that the relevant fields are not fully populated in the system, leading to unsuccessful attempts.

Fix

Import

An error was logged stating "Index was outside the bounds of the array," occurring during the metadata extraction process. The job would fail, resulting in no metadata being added back in. After successful testing, it was confirmed that removed metadata and participants are re-added to the tables upon running the QC Batch Reprocessing, indicating that the issue has been resolved.

Fix

Installation & Maintenance

Users are encountering errors when attempting to replace an archive with a repaired version from S3 or local storage during the import QC process. Although the issue has been difficult to replicate consistently, it appears that the system throws an error upon reprocessing, suggesting that this may be a lingering bug related to file replacement processes.

Fix

OCR

A bug was identified in Discovery Manager where applying templates or creating new ones sets the OCR timeout to excessively high or negative values instead of the expected 20 minutes. This issue has caused processing times to increase drastically, in some cases by up to 50 times.

Fix

Imaging

A fix was implemented to ensure the PDF imaging status remains as "Success" after completing a subsequent TIFF imaging job on the same dataset. The bar graph now accurately reflects the correct status for both PDF and TIFF imaging jobs.

Fix

Processing Job Status

A fix was implemented to ensure the job status, percent complete, and modified fields are correctly updated for all copy project settings jobs. Jobs now transition through appropriate statuses (Created, InProcess, and Complete or Error) with Percent_Complete set to 100 upon completion, and the Modified_by_user and Modified_Date fields properly populated.

Fix

Production & Export

RSMF exports were failing due to out-of-range exceptions when participant display data was enclosed solely within brackets. The export logic has been updated to handle these cases properly.

Fix

Production & Export

There was an issue where Teams data exports were failing to maintain sequential order between parent and child documents across load file chunks. This fix ensures that Teams message attachments are now exported sequentially with their parent items, preventing errors during Review Append exports.

Fix

Production & Export

In large exports following Australian Protocol, native and text file consistency has been improved. Previously, some documents, like emails and Word files, might not export fully under Australian settings. This update ensures that both Australian Protocol and Parent/Child exports now generate complete and validated files for native and text formats, improving overall export reliability, aligning with the results seen in Parent/Child exports.

Fix

Production & Export

A bug was found where the delete button for Export Templates was disabled, preventing users from deleting templates. Users had to go through a workaround by navigating to the Project Template section and returning to the Export Table for the delete option to become available.

Fix

Production & Export

An issue where the export process hangs indefinitely, and the PST files continue to grow in size without completing, suggesting repeated data entries into the PSTs. The system fails to provide clear error messages, leaving users without guidance on the issue.

Fix

Production & Export

Users are experiencing an issue where the DOCDISPLAYNAME field remains blank for non-email files during export, even though the necessary information exists in the FILENAME field. The expected mapping to the Review field "SUBJECT_OTHER" is not occurring as intended.

Fix

Project Administration

The 'Allow Duplicates' checkbox in the new uploader is not functioning correctly. When the checkbox is checked, the documents are not properly processed for duplicates. Specifically, while the upload, process, and promote jobs report success, the export job in Discovery Manager shows errors with the message ‘Error Loading Loadfile Into Review’. This error relates to an export containing 0 documents, despite the documents being promoted to Review. Additionally, the export job still shows ‘Promote to Review’ as the Deduplication setting, which enables the removal of duplicate documents, instead of ‘No Deduplication’.

Fix

Project Creation

An issue was identified where silent errors occurred in the Processing API during project creation, causing missing field mappings and subsequent upload failures.

Fix

Reports

A bug was identified where adding export file IDs to a Selective Set resulted in incorrect document and family counts being displayed, creating confusion for clients. Although the underlying data remains accurate, this visual discrepancy has been resolved to ensure that counts correctly reflect the intended items, thereby improving clarity in export operations.

Fix

*Upload Processing

Previously, The last column in the header row is not being picked up correctly during DAT file parsing, resulting in the column not being mapped properly. Now, it is displayed properly in the preview mapping heading, allowing it to be mapped correctly.