In Reveal Enterprise, special and non-alphanumeric characters may be indexed differently depending on how they appear within a term, affecting how users search for and retrieve documents. It combines findings from Reveal’s ASCII Indexing analysis and Unicode text segmentation behavior (based on Unicode Standard Annex #29). Understanding these rules ensures that search terms are structured effectively to retrieve all relevant documents.
Reveal Enterprise uses Elasticsearch, which relies on Apache Lucene and the Unicode Text Segmentation (UAX #29) standard for indexing text. This differs from legacy systems like dtSearch or SQL Full Text Indexing, which may treat punctuation and symbols differently.
In practice, this means that some special characters split words into multiple tokens, while others treat text as a single indexed term. For example, 'firstname.lastname' may be indexed as one token (single term), while '5,123.456' may be split into multiple tokens depending on the symbols used.
ASCII special characters in Reveal
Reveal Enterprise supports 6 special characters that are indexed as a single term. Only the following ASCII characters may prevent word breaks: Apostrophe ('), Dot (.), Comma (,), Colon (:), Underscore (_), and Semicolon (;).
When the special characters are enclosed within two letters, a letter and a number, or two numbers, you can expect the following results:
Pattern | Example | If the special character is one of the below, it is indexed as one term |
|---|---|---|
L.S.L | "first.last" | Single quote |
L.S.N or N.S.L | "first.123" | Underscore |
N.S.N | "5.678" | Single quote |
Searching with embedded characters
This seems rather bleak for those of us who want to find content in the language of our discovery data. We have no way of knowing how many of these wily characters may be lurking within the words, distorting the language sufficiently to evade our most diligent syntax. Add defined keyword lists, and the issue is heightened.
Here is a brief guide to search that takes the vagaries of embedded characters into account.
Use wildcards – While you may not go right the exact term in every instance, it will be within the search results. In the examples above, the typo punctuationwhile could be retrieved using punctuation* or *while, though leading wildcards are not an efficient strategy unless a last resort. Wildcards may also be used to retrieve plurals (books). Since the asterisk wildcard can represent any number of characters or none, interjecting it has no cost in the middle of a name or email address.
Proximity will not help – Proximity search is based on enumerating possible word boundaries between the terms specified. The problem here is that we have no boundaries; the text with embedded characters is seen as a single word. Sounds like a family drama, but it is all about separating words.
Practical Examples
Input Text | Indexed As | Effective Search Terms |
|---|---|---|
firstname.lastname | Firstname.lastname | Firstname.lastname |
Firstname’lastname | Firstname’lastname | Firstname’lastname |
Firstname:lastname | Firstname:lastname | Firstname:lastname |
Firstname_lastname | Firstname_lastname | Firstname_lastname |
FirstName_123 | FirstName_123 | FirstName_123 |
123’987 | 123’987 | 123’987 |
123,987 | 123,987 | 123,987 |
123.987 | 123.987 | 123.987 |
123;987 | 123;987 | 123;987 |
123_987 | 123_987 | 123_987 |
firstname@reveal.com | Firstname reveal.com | Firstname*reveal.com |
5,678.00 | 5 678.00 | 5*678.00 |
Joyce’sFinnegan’sWake | Joyces’finnegan’swake | Joyce’finnegan’swake |
Section-3 | section 3 | Section*3 |
Indexing and search guidance
Character | Indexed as Part of Word | Recommended Search Strategy |
|---|---|---|
' | Yes | Use wildcard (e.g., joyce*finnegan) |
. | Yes | Include wildcard (e.g., example*com) |
, | In numeric contexts | Use numeric wildcard (e.g., 5*678) |
: | Sometimes | Search with/without symbol |
_ | Always | Search exact term or wildcard |
; | In numeric sequences | Include wildcard around it |
All others | No | Use separate search terms |