Reveal Query Language

Details on the syntax and use of RQL in Reveal

Required Query Features

In addressing all the features needed in the Reveal Query Language (RQL), an attempt is being made to remain backwards compatible with Review 10’s query language. Here are the known places where RQL is not backwards compatible with the Review 10 query language:

1. In RQL you must use quotes for phrases. Review 10 assumed that a sequence of terms was a phrase, while the initial RQL implementation will call that an error.

2. Review 10 supports both w/3 and /3/ as syntax for proximity queries. RQL only supports w/3.

3. Review 10 supports ?, !, and as wildcard characters. The initial RQL release will only support , and will interpret * as zero or more characters.

4. Special characters can make the index interpret what looks like a single word, for example “AT&T”, as a phrase, so RQL users will need to put quotes around these terms to specify the correct phrase search behavior. Please note, Reveal ElasticSearch does not support searching of individual special characters, e.g., "%". A deeper discussion of handling special characters in RQL may be found in Search Guidance – Unicode Rules for Indexing. If your project data has been processed via Reveal Processing Discovery Manager, there are other capabilities utilizing its dtSearch engine for searching special characters. More information can be found here: https://processing-help.revealdata.com/en/APPENDIX-F---dtSearch-Syntax-Guide.html.

5. The Reveal 10 documentation talks about only supporting binary NOT, and specifically says it does not support AND NOT. To keep the binary NOT from being ambiguous, RQL supports A AND NOT B, and A OR NOT B. RQL does not support A NOT B, nor does it support A NOT W/N B.

6. Common words will be treated differently in Reveal and RQL. Instead of using the StopFilter to keep common words from being indexed, the Elasticsearch CommonGramsFilter will be used instead to support efficient searching of common words. This eliminates most of the edge cases associated with common words appearing in queries.

The following tables describe features of the query language.

Operator precedence is supported to maintain backwards compatibility with Review’s prior query language.

Note

RQL will validate keyword search syntax before submitting a search. It will parse the query as entered in the search bar to identify any errors and notify the user to examine and correct the syntax before submitting.

Special RQL Field Names and Search Profiles

Reveal allows users to create search profiles, which are lists of fields to search. When a search profile is used with an RQL query, any query term that does not specify a field name will search the fields specified in the search profile. RQL also supports some special field names that can be used to conveniently search multiple fields and whose behavior won’t change if a search profile is used.

• BODY:

  Targets all fields containing the main document text content. This includes Extracted, OCR, Loaded Text, as well as other text sets indexed into Elastic, such as translation, transcription, and newly generated text sets. If a user adds another text set and populates it with text, that will also contribute to the BODY search.

• METADATA:

  Searches only the specific text-based metadata fields, excluding body content. This includes fields like folders, BRS-generated fields, tags, keywords, and communication analysis fields. It is focused on document-specific attributes rather than the actual content of the document.

• ALL:

  Performs a broad search across both BODY text and all searchable METADATA fields. This search targets both the main document content and the associated metadata, providing a full scope of searchable document information.

In the 2024.11 release, the Native/HTML view will be deprecated, so it's recommended not to focus on HTML/Native text sets. The Extracted view will continue to contain the document text but without the HTML tags present in the Native/HTML view.

When no field name is specified, the default behavior is to search the fields that the special field name BODY searches. When no field name is specified and a search profile is used, the search profile fields are used.

Search profiles can only contain text and body text fields.

Where field names are used in an RQL search, they must be the TABLE names (no spaces or dashes, underscores allowed). These field names may be looked up and selected using the RQL Helper’s Insert Fields… link under Advanced Keyword Search; a tilde (~) character may also be used to open the field list.

RQL Description

The intent is that these operators can be combined to create interesting query expressions. This includes arbitrarily nested parentheses. How deep “arbitrarily nested” goes is negotiable.

Date, Time, and Date-Time Queries

The details of how Date, Time, and Date-Time fields are indexed ultimately determine what results you will get from the date and time queries RQL supports.

How Date, Time, and Date-Time are Indexed

All of these fields are indexed as milliseconds since epoch, and each value is represented as a single specific millisecond. Here is what you need to know to understand how each of these fields are indexed:

• Epoch is midnight (12am) January 1, 1970 UTC (in milliseconds since epoch this is 0).

• Dates and times prior to midnight January 1, 1970 UTC have a negative value for milliseconds since epoch.

• If only a date is supplied, the time is assumed to be midnight (00:00:00.000).

• If only a time is supplied, the date is assumed to be January 1, 1970 (1970-01-01).

The following table has some examples.

Querying Date and Date-Time Fields

Currently RQL treats Date and Date-Time field queries the same. It is up to the user to understand that dates from a Date field are treated as Date-Time fields where the time is always midnight.

Let’s look at some queries and see which queries find 10 June 2022 (2022-06-10), 11 June 2022 (2022-06-11), or 12 June 2022 (2022-06-12).

Here is what happens in the above queries:

1. The RQL query MASTERDATE::2022-06-11 searches for date-time values that fall anywhere on the day 2022-06-11. The query finds date fields that contained 2022-06-11 and does not find date fields that contained 2022-06-10 or 2022-06-12.

2. The RQL query MASTERDATE >= 2022-06-10 AND MASTERDATE <= 2022-06-12 searches for date-time values that fall anywhere on the days 2022-06-10, 2022-06-11, and 2022-06-12. The date fields containing 2022-06-10, 2022-06-11, or 2022-06-12 are found.

3. The RQL query MASTERDATE > 2022-06-11T00:00:00 AND MASTERDATE < 2022-06-12T00:00:00 includes all but one second of 2022-06-11. Since that excluded second is midnight 2022-06-11 and since no other midnights are included in this query, no date values are found.

Querying Time Fields

In RQL it is illegal to query a time field with date information. Even though times are treated as if they are date-time fields where the date is 1970-01-01, in RQL you will never specify this date when searching a time field. Here are some examples:

Expansion Queries using the EXPAND operator

While the initial release of Reveal supports expanding query results to include families, threads, etc., having queries with multiple steps of expanding results, searching within that expansion, then expanding those results, is not easy to do.

Why is RQL support for expansions needed?

Here is an example of a series of searches that includes four expansions of the results:

1. Perform a search.

2. Expand to families — adds attachments and/or messages.

3. Restrict to Excel files — step 2 likely added Excel files not found in step 1.

4. Expand to families — results in only families with at least 1 Excel file.

5. Search results of step 4 to find documents out of that set that are marked responsive.

6. Expand to families — results in families with at least 1 Excel file and at least one family member marked responsive.

7. Search within results of step 6 for documents that are not marked responsive.

8. Expand to families — results in families with at least 1 Excel file, at least one family member marked responsive, and at least one family member marked non-responsive.

To do the above without RQL support for expansions, every time you want to search within the expanded results, you would have to first save the expanded results to a folder, so you could start a new search that searches within that folder.

Instead of running a single query, you would have to run four separate queries, creating 3 different folders to hold the temporary results.

If you want to rerun the queries later to see how document additions and more review has changed the results, you again need to run four separate queries and create three new folders. Adding RQL support makes running and rerunning a query like this much easier and potentially more performant.

The EXPAND operator

EXPAND is a unary operator with syntax similar to NOT. One big difference is that you have to specify what fields to use for the expansion.

For this discussion of how the EXPAND operator works, assume that your dataset has the following fields:

What the EXPAND operator looks like

The EXPAND operator itself must include the fields that will be used for the expansion. Here are some examples and their meanings:

• EXPAND{FAMILYID} — Expand to include all family members based on the FamilyID field

• EXPAND{FAMILYID, THREADID} — Expand to include all family members and thread members based on FamilyID and THREADID. (NOTE - threads include family members, so you could just EXPAND{THREADID}.)

• EXPAND{NEARDUPSETID, EXACTDUPSETID} — Expand to include both the near duplicates and exact duplicates of the query’s results

• EXPAND{NEARDUPSETID, THREADID} — NOTE - both expansions are based on the set of results returned by the query expression this operator is applied to. Near duplicates that are in threads that aren’t in the original set of results won’t expand to include their thread members. You would have to say EXPAND{THREADID} EXPAND{NEARDUPSETID} <expr> to include the threads of all the near duplicate documents.

EXPAND examples

Simple Expansion

EXPAND{FAMILYID} enron

Search for enron then expand the results

Multi-Field Less Simple Expansion

EXPAND{NEARDUPSETID, THREADID} (enron AND gas)

Search for enron AND gas then add near duplicates of search results and thread members of search results.

Nested Expansions

EXPAND{THREADID} EXPAND{NEARDUPSETID} (Enron AND gas)

Search for Enron AND gas then expand to include near duplicates of the results. Finally expand to include the thread members of both the original results and the near duplicates of the original results.

Recreating the Original Example

Here are the fields that this example assumes are in the dataset for this example:

Earlier the following example was provided as motivation for providing RQL syntax for query expansions:

1. Perform a search.

2. Expand to families — adds attachments and/or messages.

3. Restrict to Excel files — step 2 likely added Excel files not found in step 1.

4. Expand to families — results in only families with at least 1 Excel file.

5. Search results of step 4 to find documents out of that set that are marked responsive.

6. Expand to families — results in families with at least 1 Excel file and at least one family member marked responsive.

7. Search within results of step 6 for documents that are not marked responsive.

8. Expand to families — results in families with at least 1 Excel file, at least one family member marked responsive, and at least one family member marked non-responsive.

Here is how the RQL could be built up to finally produce a single query expression:

1. enron AND gas — an arbitrary query for this example.

2. EXPAND{FAMILYID} (enron AND gas) — original query expanded to include families.

3. (EXTENSION::xls OR EXTENSION::xlsx) AND EXPAND{FAMILYID} (enron AND gas) — restrict to Excel files.

4. EXPAND{FAMILYID} ((EXTENSION::xls OR EXTENSION::xlsx) AND EXPAND{FAMILYID} (enron AND gas)) — families with Excel files.

5. RESPONSIVE::true AND EXPAND{FAMILYID} ((EXTENSION::xls OR EXTENSION::xlsx) AND EXPAND{FAMILYID} (enron AND gas)) — responsive documents from (4).

6. EXPAND{FAMILYID} (RESPONSIVE::true AND EXPAND{FAMILYID} ((EXTENSION::xls OR EXTENSION::xlsx) AND EXPAND{FAMILYID} (enron AND gas))) — families with at least 1 Excel file and 1 responsive document.

7. RESPONSIVE::false AND EXPAND{FAMILYID} (RESPONSIVE::true AND EXPAND{FAMILYID} ((EXTENSION::xls OR EXTENSION::xlsx) AND EXPAND{FAMILYID} (enron AND gas))) — non-responsive documents from (6).

8. EXPAND{FAMILYID} (RESPONSIVE::false AND EXPAND{FAMILYID} (RESPONSIVE::true AND EXPAND{FAMILYID} ((EXTENSION::xls OR EXTENSION::xlsx) AND EXPAND{FAMILYID} (enron AND gas)))) — families with at least 1 Excel file, 1 responsive document, and 1 non-responsive document.

The nesting of the queries gets complex once we get to step 8, but what used to have to be run as four separate queries in Reveal is now a single RQL query that can be saved and rerun later to see how the results have changed.