Understanding and Resolving Saved Searches Impacted by Upgrade
What are the changes?
We are excited to announce the recent upgrade of our underlying search engine, which provides an even better Logikcull search experience.
While more than 97% of all Saved Searches remain unaffected, there are a few minor changes that you should be aware of. The changes between the old and new search engines are minimal, and less than 3% of saved searches are impacted.
In this article, we'll review these impacts and provide steps to ensure your saved searches work as expected.
We have locked impacted Searches for you
To ensure that the output of your saved searches is preserved, we have automatically locked impacted saved searches that have results with outputs of less than 1 Million documents.
The two potential impact types (if impacted at all):
There are two main types of Saved Search impacts as a result of this upgrade:
Search Result Variances: Queries will run, but the search results may contain some variances from results returned before the upgrade.
Search Errors: Queries will no longer run and display an error message.
Identifying Impacted Saved Searches:
If your project contains any impacted saved searches, we have highlighted Saved Searches that were impacted, and the type of impact, within the Saved Search list. If your project does not contain any impacted saved searches, your saved search list will look the same as before the upgrade.
Two new columns have been temporarily added to the Saved Search table for Projects with impacted saved searches, Search Status, and Impact Details. If you do not see these columns within your project's saved search list, then that means there were no searches impacted within that project.
The Search Status column provides a quick way to determine if a saved search has been impacted. A new Search Status filter allows you to view saved searches easily based on their status.
The Impact Details column provides additional dynamic information on saved searches with an impact.
Search Status | Definition |
Green “Available” | Saved searches that were not forced locked and can be run |
Red “Non-updating query” | Save Search has errors, but we were able to lock the saved search so that you can still run the search and see the results returned before the upgrade. This search will need to be recreated with adjusted syntax if the output needs to provide updated/dynamic results. (See below for syntax changes) |
Yellow “Outputs may vary” | Saved Search result output may return slightly different results than before the upgrade, so we locked the search results to preserve the output. Recreate the saved search and compare the output with the locked results to identify the exact document differences. Saved searches with output variances that could not be locked (has an output of >1M documents) will appear green and available as they still can be run. |
Grey “Unavailable - Archived” | The saved search has errors, but we could not lock the results output as it exceeded 1M documents, so it can no longer be run. You will need to create a new search with revised syntax if the search is still needed. (See Search Errors below for syntax update tips) |
Search Result Variances (Outputs may vary)
This upgrade introduces several advancements, including improved search performance, enhanced security features, and more sophisticated analytical capabilities. In a few cases, these changes may cause saved searches to return slightly different document outputs than before the upgrade. When that happens, we will automatically lock the search for you so that you can still see the same set of documents returned before the upgrade.
Most of the time, search result variances from the old search engine to the new search engine are due to the improvements within the new search engine that typically result in more accurate results, which can be a mix of reducing irrelevant results and increasing found relevant results.
In general, there are three types of variances:
Searches that produce more results than those produced before the upgrade
Searches that produce fewer results than those produced before the upgrade
Searches that produce different results than those produced before the upgrade
What causes search result variances/output differences?
Updated Algorithms: Elasticsearch version 8.14 includes updates to the underlying algorithms that process search queries. These changes are designed to improve the accuracy and relevancy of the results. However, they may also lead to different outcomes for the same queries run on the older version.
Scoring Adjustments: Elasticsearch has refined its relevance score calculation process. The scoring adjustments help prioritize more relevant documents differently than before, which might alter the order and presence of documents in your search results.
New Analytical Functions: With additional analytical functions, the system can interpret and process your data more precisely, potentially changing the results for complex queries.
Index Changes: Changes in indexing strategies and configurations to improve search performance and data handling can also influence the results of your queries.
How to identify exact document output differences
To identify exact document variances for saved searches that were locked before the migration, you can run the locked output, apply a batch tag to the output, recreate and rerun the search, and compare the outputs from the first saved search to the second saved search. The easiest way to do this is to select the unlock option within the three dots dropdown, check the “Keep a copy with locked search results" option, and click the "Create Unlocked Copy" button. This will create an unlocked copy of your saved search so that you can compare the differences by applying batch tags.
Search Errors (Non-updating query)
Two Types of Search Errors
Overly Complex Nested Searches
Maximum Number of Supported Simultaneous Criteria
Overly Complex Nested Searches
Some search errors are caused by excessive nested searches that can lead to high computational overhead and diminishing search performance and efficiency.
Query update tips:
Simplify Nested Searches: Break down complex nested queries into simpler components.
Limit the Depth of Nesting: Avoid deeply nested queries.
Examples:
Instead of:
(status:active AND (category:electronics OR category:appliances) AND (brand:Samsung OR brand:LG)) Break into two queries and use:
Query 1:
(status:active AND (category:electronics OR category:appliances) AND brand:Samsung)Query 2:
(status:active AND (category:electronics OR category:appliances) AND brand:LG) Maximum Number of Supported Simultaneous Criteria
Some search errors are caused by queries exceeding 78K different simultaneous criteria/clauses, which is more commonly caused by an overuse and/or overly inclusive wildcards.
Query update tips:
Avoid Leading Wildcards: Queries like [*ing] are discouraged as they would return every word that ends with [ing], which would likely introduce a lot of noise in your results. Instead, consider more specific terms and/or use a prefix query like [manag*]
Limit Over-Use of Wildcards: To ensure your searches run and produce relevant results, use wildcards sparingly and only when necessary. For example, change [*fact*] to [manufact* OR refact* OR benificat* ].
Reduce the Number of Clauses: Break down queries with many criteria into smaller, more manageable queries.
Use Filters Where Possible: Utilize filters to narrow down results before applying multiple criteria.
Examples:
Instead of:
[field1:value1 AND field2:value2 AND field3:value3 AND field4:value4] Use multiple searches:
[field1:value1 AND field2:value2] Then after running the first search, search for
[field3:value3 AND field4:value4] Frequently Asked Questions:
Q: What will happen to any jobs or integrations actively syncing at the time of the upgrade?
A: Jobs in progress during the upgrade will be paused and resumed after the upgrade.
Q: For Searches that will result in different outputs, how can I tell which documents are now being returned but previously were not?
A: If the Saved Search output was less than 1 million documents, we automatically locked the search for you. To see exact differences in outputted documents, select the unlock option, check the “Keep a copy with locked search results" option, and click the "Create Unlocked Copy" button. This will create an unlocked copy of your saved search so that you can compare the differences by applying batch tags.