Error Resolution Troubleshooting Table

ERROR GraphResource Exception while writing Graph object to output stream

Possible issue accessing target drive.

Verify that drive volumes are mounted, accessible, and not full.

See System Health Checks

ERROR AsyncProcess Process com.purediscovery.brainspace.exception. BDFatalException: Failed to execute FocusProcess.

Possible issue accessing target drive.

Check analysis-service and /localdata drive on the On-Demand Analytics server.

See System Health Checks

ERROR – what(): Error writing to file: /data/brainspace/builds/hash/ part/workingBrainLibrary/Root/ working/doc_output.bin

Destination directory may be not mounted or directory may be full and unable to write to.

Check drive is mounted properly and not full. Note: /data usage will increase during term expansion phase of the build, then cleans up once the brain is built and index complete. Make sure /data has sufficient space for expansion during build process.

See System Health Checks

Unable to call server stats URL: https://x.x.x.x:8081/rest/analysisstats?type=RUNTIME. Reason: null” and “Unhandled exception: Directory: /data/brainspace does NOT exist"

The /data directory is unavailable, possibly due to Application server being started before Analytics server and NFS was fully started and available.

Check if the /data/brainspace directory is mounted.

If not, getting it mounted should resolve the issue based on the error.

See System Health Checks.

ERROR UpdateServerStatsTask Could not get system stats from: [analytics_server]:1604. java.net.UnknownHostException: [analytics_server]. Continuing… UpdateServerStatsTask Could not get system stats from: [server_name].ProcessingException: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative names matching IP address xx.xx.xx.xx found.

Loss of communication to server. Check for server outage or network maintenance corresponding to the time stamp on the error.

Ensure all servers are up and healthy and check network connectivity between Brainspace servers.

An I/O error has occurred while writing a response message entity to the container output stream org.glassfish.jersey.server.Server Runtime$Responder. writeResponse java.io.IOException: Connection reset by peer.

Client’s connection was terminated prematurely. Often this is a temporary condition.

Check communication path between hosts and retry. If this error frequently occurs, consider increasing the HAProxy timeout setting.

No subject alternative names present.

According to SO: v3.ext authorityKeyIdentifier=keyid, issuer basicConstraints=CA:FALSE keyUsage = digitalSignature, nonRepudiation, keyEncipherment,

dataEncipherment subjectAltName = @alt_names [alt_names] DNS.1 = %%DOMAIN%%

Supply an extra parameter to openssl when you are creating the cert, basically -sha256 -extfile v3.ext where v3.ext is a file like so, with %%DOMAIN%% replaced with the same name you use

as your Common Name. Typically you would set the Common Name and %%DOMAIN%% to the domain you are trying to generate a cert for. So if it was www.mysupersite.com then you would use that for both.

Caused by: java.lang.OutOfMemoryError: unable to create new native thread Caused by: java.util.concurrent.ExecutionException: java.lang.OutOfMemoryError: Java heap space Caused by: java.lang.OutOfMemoryError: Java heap space

Tomcat ran out of memory (or analysisserver on Analytics/OD servers).

This error is caused by a lack of memory on the box, not because of a lack of heap space (which has a different signature). The error indicates that the box is oversubscribed / underpowered.

If this recurs, please run: kill -s 3 <tomcat pid> and save off the catalina.out file as part of the ticket. If this occurs more than once, increase memory in the server.

ERROR ContextRunnable implementation [class com.purediscovery.brainspace.dataset .DatasetInitializer$4] threw an exception.com.purediscovery. brainspace.exception. BDApplicationException: Could not copy output directory to dataset. Caused by: java.io.IOException: No space left on device

No space left on device. Likely the storage drive is full and no further data can be written. Look further in log, in ERROR and Caused by entries for specific disk volume affected.

Delete datasets (from UI; deleting direct from disk will cause database discrepancy and data corruption), unneeded old log or backup files, or expand the disk with additional storage.

WARN AuthenticationFilter Detected possible session hijack attempt! Request IP: ‘TCP/IP: 62.190.57.100’ Original IP: ‘TCP/IP: 62.190.57.96’ Request User-Agent: ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36’ Original User-Agent: ‘Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169

Safari/537.36’ Affected session userId: 44a006b7-44c6-412a-a509-c3494e4a9726

If the user’s source IP changes, it looks like possible hijack attempt.

Network administrator should investigate and confirm that it is not due to a Brainspace problem.

Server returned an error: target /localdata-share/datasets/<BuildHash>/token-tdm is not a directory or file

/localdata-share not mounted

Check that /localdata-share is mounted on the ODA, mount it if it is not.

See System Health Checks

Server returned an error: target /localdata-share/datasets/<BuildHash>/tdm-extended is not a directory or file

/localdata-share not mounted

Check that /localdata-share is mounted on the ODA, mount it if it is not.

See System Health Checks

ERROR ActionService Tagging.getTagChoiceSummaryCounts failed with unhandled exception.

N/A

Check that Relativity^® tags are properly set.

connector.activity.RetriableSyncCall.login AndRetry(RetriableSyncCall. java:106) Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The TCP/IP connection to the host Relativity_SP, port 1433 has failed. Error: “Relativity_SP: Name or service not known. Verify the connection properties. Make sure that an instance of SQL Server is running on the host and accepting TCP/IP connections at the port. Make sure that TCP connections to the port are not blocked by a firewall.” SQLServerException.make FromDriverError. ConvertConnectException ToSQLServerException

The server cannot resolve the SQL host name provided in the connector setup, or traffic to Relativity^® SQL host is blocked on port 1433.

Verify that DNS and/or hosts file includes proper server name/IP info. Verify that the network path is open.

com.microsoft.sqlserver.jdbc. SQLServerException.makeFrom DriverError(SQLServer Exception.java:227)

com.microsoft.sqlserver.jdbc. SQLServerException.Convert ConnectExceptionToSQL ServerException(SQLServer Exception.java:2

This error normally means that server cannot resolve the sql host name provided in the connector setup. Or traffic to Relativity^® SQL host is blocked on port 1433.

Verify DNS and/or hosts file includes proper server name/IP info. Verify network path is open.

ERROR: Failed to ensure access to the dataSource for the updated Ingest.

Due to moved workspace that was originally built to a “new” Rel instance… this then changes the artifact id of the workspace... and we are unable to find what we are looking for.

N/A

com.brainspace.plugin.Connector CommunicationException: either the fieldDisplayName [Field Example] or the idFieldDisplayName [DocID] could not be found f[null]

Brainspace is unable to see the saved search within Relativity^®, so reports back that fields have come back as null.

Based on the official Relativity^® Guide for 9.6 (applies to 9.5), the below parameter is what you want to change: “PDVDefaultQuery CacheSize”. The default value is 1000, but you’ll want to increase this until you can see your additional saved searches.

Caused by: com.microsoft.sqlserver.jdbc .SQLServerException: Connection reset

Relativity^® source caused reset/network timeout.

Restart session to recover.

Caused by: com.brainspace plugin.exception.ConnectorLogin Exception: REST call returned code: 401, msg=Unauthorized

Connector credentials do not have access to target workspace.

Clear credentials (from profile, clear credentials associated with the connector, retry and provide updated credentials.

ERROR RelativityDB Should not happen: Encountered two Relativity^® fields of the same name for the same artifact

Relativity^® introduced a new field in the production scheme that sometimes causes conflicts. Production Begin Bates/Production::Begin Bates and Production End Bates/Production::End Bates. SQL strips out the :: and views it as the same name.

This issue must be corrected in Relativity^® by making the fields unique and avoid using the double-colon (::).

ERROR DBAccessDefaults PSQLException encountered with SQLState Caused by: org.postgresql.util PSQLException: Connection to localhost:5432 refused DBConnection Query aborted

Not able to communicate with Postgres Database. Either the database is not running or a local firewall service may have started and be blocking the port (5432).

Check the Postgres service is running. Verify local and network firewalls allow proper access.

Error: Caused by: com.microsoft.sqlserver.jdbc .SQLServerException: Cannot open database “EDDS requested by the login. The login failed. ClientConnectionId: xxx

Error trying to access Relativity^®. This could cause field mapping issues, overlay failures, etc.

Check health and access to Relativity^®.

ERROR  RelativityPlusResponseHandler Call to [https://<RelURL>.com:443/Relativity.REST/api/ Relativity.Objects/workspace/<ID>/object/queryslim] Failed: HTTP/1.1 503 Service Unavailable

This Error is being thrown by Relativity^® and Brainspace is unable to retrieve the requested information.

This error typically indicates a network related issue between Brainspace and Relativity^®, recommendation is to check for DNS changes. To test/verify without Brainspace run the following from the command line on the Application Server: curl -vvv -ks https://< Relativity_base_URL>/Relativity/Identity/connect/token.

[AsyncProcess-##] ERROR AsyncProcessBase Process [IngestProcess{ingestProcessData= IngestProcessData{datasetId='##', idField='<ID Field>', userId=<userID_GUID>, ingestId=1}}] Exception occurred in phase: WORK. Time: 36246911mscom.purediscovery. brainspace.exception.BDPluginLoginException: No credentials available for connector.

This error is generated when the OAuth2 token for Relativity^® has expired.

Re-Authenticate with Relativity^® by going into Brainspace and going to “Profile”. Scroll down to “Third Party Authentication” and click “Re-Authenticate”.

ERROR PersonResource construct (personLabels, fields) Exception occurred while trying to construct a PersonLabels with limited fields.

N/A

Check the dataset Alias report.

Error creating ZipSpliterator.

File password protected; could not be read.

Unlock file prior to ingestion.

Unable to process: document is encrypted.

File encrypted.

Decrypt prior to ingestion.

Document mime-type not in whitelist.

Native ingest not supported.

Add required file type to whitelist. WARNING: Some file formats may produce insufficient formatting and unexpected results.

AsyncProcessBase Process [BuildProcess[build: dataset_name]] Exception occurred in phase: WORK.

Indicates there is something in the data which is not being allowed in the application. Please verify the dat file coding inside is UTF-8 and not have any UTF-16 or anything.

N/A

emt errors – Warning: key XXXXXXXX-00012345 has a parentid XXXXXXXX-00012567 that does not exist

Incorrect use of Parent ID, Family ID or Group ID. Or parent document does not exist in saved search results from Relativity^®.

N/A

Error com.purediscovery.exception.ChildProcess Exception: rsvd returned error code 134

Problem with parent-child relationships.

Check use of parent ID and family ID. Typically one or the other of these is used, but not both. Verify metadata format for conversation ID.

ERROR BDApplicationExceptionMapper Handling raw BDApplicationException:

Indication of several other errors. Look for additional detail in logs.

N/A

ERROR dataset could not load graph data at /localdata/brainspace/…/graph-data. Allowing dataset to load without this capability.com.purediscovery.exception. PDException: ccLength file is unexpected size.

Discovery 5.4 not able to handle exceeded field.

N/A

ERROR PermsDB Unable to retrieve permissions for user: 3ERROR ServerAuthenticationHelper Exception occurred while trying to authenticate request […/session/current].

Due to user session logged out.

Log out and then log back in to Brainspace.

ERROR DataSetManager could not initialize dataset BDLicenseException: License load error: Found 2 license files found in /var/lib/brains/.brainspace. Only one is needed.

Multiple license files exist in required directory.

Ensure only one valid license file exists in /var/lib/brains/.brainspace. You may need to replace the old file then restart services (tomcat on app; analysis-service on An/OD).

INFO – License error (/var/lib/brains/.brainspace/brainspace_license.txt): No licensed MAC addresses [04:4c:8d:6e:d9:b8] found in [06:5c:45:ef:0b:a0]

Most likely the MAC address(es) changed on client servers. This is common at cloud VM providers.

Provide new MAC addresses to CSM and request new license be provided. Also, set MAC not to change upon VM reboot to avoid this problem.

Error during install/upgrade: generating certificate and configuring SSL mode. keytool error: java.lang.Exception: Key pair not generated, alias discovery already exists.

There is a jks file under /var/lib/brains/.brainspace folder from a previous install.

N/A

java.lang.IllegalArgumentException: All readers must have same maxDoc: 2224519!=2131289.

Tomcat was most likely interrupted when shutting down and the CMML Indexes were not successfully written, (and subsequently a Dataset is not starting)

The CMML Indexes will need to be rebuilt. Note – no work product will be lost. See Force CMML Indexes to be Rebuilt.

buildAuditLog is damaged. You won’t be able to process data until your audit log is repaired. Caused by: com.purediscovery.exception. PDException: Can’t commit changes to a read only log.

Build audit log is corrupted; this could be caused by filled up disk – please verify drive volumes and free space available.

Remove the old /var/lib/brains/.brainspace/buildAuditLog file. Replace the license with a new one (from Brainspace Support). With the new license file in place, run bt create-audit to re-create the audit log to resolve the issue.

ERROR ServerProperties File not found at: /root/.brainspace/brainspace. defaults.properties [localhost-startStop-1] ERROR ServerProperties File not found at: /root/.brainspace/brainspace. properties

Indicates that the service was not started by the brains user or the brains user .bash_profile was not sourced. Did an Admin recently login and access application files as user other than brains? Or new license applied by other than brains user? Possibly the tomcat service was manually started as the brains user but the .bash_profile was not sourced and resources not set properly.

The Sysv init scripts / systemd unit files should be used instead of manually starting tomcat via startup.sh

LockedTransactionFile Lock file /data/brainspace/builds/<BuildHash>/accessLog/lock.txnlog already exists.

Lock file can occur if system, storage, servers, etc. become unavailable during normal Brainspace operations. Lock files are common. They prevent conflicting updates to a database table or field. If the process that clears the lock gets interrupted, the lock file may need to be manually cleared (deleted).

For example, if 964e1f41-3b78-4ba1-af4f-69b2f32505eb is the unique identifier for the dataset, Brainspace keeps track of running builds by placing a lock file in each build’s corresponding directory (inside the accessLog directory) named “lock.txnlog.” This file keeps other sessions from running a build on the same dataset and essentially stepping on the active build session.

If the PID is not running on the application server, then you can delete the lock file. If the PID is still running, please reference the build info log for the build. Follow the instructions in the log precisely. Verify the PID inside of lock.txnlog is not still running. You can simply wait 10-15 minutes if services are running and the lock should clear. Or ensure the build/overlay process is stopped or cancelled. To check for PID and service status: cat /data/brainspace/builds /<buildhash>/accessLog /lock.txnlog ps aux | grep <pid from lock file>

Unable to retrieve user info

Message observed on login screen (UI). Postgres database is down or not accessible.

Make sure Postgres database was started properly after server maintenance or outage and before starting Brainspace application services.

Startup order found here

DEBUG IngestService Overriding auto-overlay flag because this connector doesn’t support it or has no overlay fields.

When the Overlay option is turned on after the Relativity^® connector is created, it turns off after the dataset is saved.

When creating a Relativity^® connector, make sure there are fields selected to overlay in the available fields section. Click and do a Ctrl+A to see all options. Save the connector, and then the option will be available to overlay.

ERROR ServerAuthenticationHelper API Key is required but not present in request.

Non-Impacting

Ignore; Brainspace Development to eliminate.

ERROR UserProvider No principal (User or API key) found in request.

Non-Impacting

Ignore; Brainspace Development to eliminate.

ERROR AtmosphereHandlerPubSub Unable to process incoming message: {“type”:”SUBSCRIBE”, “channel”:”uri:dataset-queue”}

Non-Impacting

Ignore; Brainspace Development to eliminate.

ERROR SetSessionListener Set service for dataset: 15 is NULL, has the dataset been shutdown?

Non-Impacting

Ignore; Brainspace Development to eliminate.

ERROR – terminate called after throwing an instance of ‘std::runtime_error’

N/A

N/A

ERROR BDUnauthorizedAccess ExceptionMappercom.purediscovery.rest. auth. LoginResource.login Login Failed com.purediscovery.brainspace .exception .BDUnauthorizedAccessException: Login Failed

N/A

N/A