Typo is something that often happens and can reduce user’s experience, fortunately, Elasticsearch can handle it easily with Fuzzy Query. Handling typos is a must if you’re building an advanced autocomplete system with Elasticsearch. If you want to create a simple one instead, you can read my other articles “ Create a Simple Autocomplete With Elasticsearch “. What is fuzzy logic Fuzzy logic is a mathematics logic in which the truth of variables might be any number between 0 and 1. It is different with a Boolean logic that only has the truth values either 0 or 1. In Elasticsearch, fuzzy query means the terms in the queries don’t have to be the exact match with the terms in the Inverted Index. To calculate the distance between query, Elasticsearch uses . Levenshtein Distance Algorithm How to calculate distance using Levenshtein Distance Algorithm Calculating a distance with Levenshtein Distance Algorithm is easy. You just need to compare the first and second word character by character. If the character is different, then you can add the distance between the words by one. Let’s see an example, how to calculate the distance between the common typo word “Gppgle” with the correct word “Google”. After we calculate the distance between “Gppgle” and “Google” with Levenshtein Distance Algorithm, we can see that the distance is 2. Fuzzy Query in Elasticsearch Handling typo in Elasticsearch with Fuzzy Query is also simple. Let’s start with making an example of the typo word “Gppgle”. Request When we’re using normal Match Query, the Elasticsearch will analyze the query "gppgle" first before searching it into the Elasticsearch. The only term in the inverted index is “google” and it doesn’t match the term "gppgle". Therefore, the Elasticsearch won't return any result. Now, let’s try Elasticsearch’s fuzzy in Match Query. As you can see, with fuzzy, the Elasticsearch returned a response. We’ve learnt in the before that "gppgle" and "google" have the distance of 2. In the query, we inserted instead of a number. Why is it working? "fuzziness":"AUTO" Elasticsearch will determine what fuzziness distance is appropriate if we use value in the field. "AUTO" "fuzziness" For 6 characters, the Elasticsearch by default will allow 2 edit distance. fuzziness is preferable, but you can tune it with an exact number if you want to. "AUTO" Now, let’s try with an exact number to prove that "gppgle" and "google" have a distance of 2. When we use , no result is returned by the Elasticsearch. "fuzziness":"1" With , though, the Elasticsearch returned the document "google". "fuzziness":"2" This proves our previous distance calculation of "gppgle" and "google" with Levenshtein Distance Algorithm, in which the result is 2. Two types of a fuzzy query in Elasticsearch In the previous example, we use a fuzzy query as a parameter inside Match Query. But there is another way to use the fuzzy feature, Fuzzy Query. Seems to be the same! So, what’s the difference between them? Fuzzy Query Fuzzy Query works like just Term Query, the query to Elasticsearch is not analyzed and used raw to search the Inverted Index. For example, let’s index one more document "Hong Kong" . Let’s look on what terms the analyzer produces with Elasticsearch’s Analyze API. As you can see, the produce two terms, "hong" and "kong". standard_analyzer If you read my other article “ ”, you’d know that if we use a term query to search "Hong Kong" then we won't get any result. Elasticsearch: Text vs. Keyword This is because there is no term that has less than 2 edit distance with "Hong Kong" in the Elasticsearch. Now, Let’s try Fuzzy Query with "Hpng" . Term "Hpng" in the query and the term "hong" in the Elasticsearch have a distance of two. Remember that the term queried and the term in the inverted index is case-sensitive, the distance “2” comes from the difference between "Hp" and "ho". Match Query with Fuzziness parameter Match Query with fuzziness parameter is more preferable than Fuzzy Query. The analyzer in the query will analyze your query before searching it into the Inverted Index. Let’s try the same query as we did in the Fuzzy Query’s section. As expected, both queries returned a result! The first query, "Hpng Kong" is analyzed into "hpng" and "kong". Both terms "hpng" and "kong" exist in the Inverted Index. "hpng" and "hong" matched with a distance of 1. While "kong" and "kong" match perfectly. One thing to note if you plan to use Match Query is that every of the terms in the query will allow fuzziness. We can try querying with "hggg kggg" which has an edit distance of 4 with "Hong Kong" using . "fuzziness":2 Tuning the Fuzzy Query in Elasticsearch You can tune the Fuzzy Query to match your use case. In this section, I will write about the parameters that we can change in the query. Fuzziness Fuzziness is the heart of Fuzzy Query. The value that we pass to this parameter is the maximum distance allowed. There are two types of value that we can pass, an integer for exact maximum distance and . "AUTO" The value allows the fuzziness in the query to be dynamic. "AUTO" We can tune 2 parameters in the value and write it as . The query will set fuzziness as 0 if the term length is below the low value. If the term length is between the low and high value, the query will set the fuzziness to 1. Last, If the term length is more than the high value, the query will set the fuzziness to 2. "AUTO" "AUTO:[low],[high]" The Elasticsearch will use 3 and 6 as the default if the low and high value is not determined. Let’s use an example with a document “Fuzzy Query in Elasticsearch allows you to handle typos”. We can try some queries to prove the mechanism of we described earlier. AUTO “tp”: 1 edit distance from “to”. “Fyzzy”: 1 edit distance from “Fuzzy”. “Fyzyy”: 2 edit distance from “Fuzzy”. “Elastissearcc”: 2 edit distance from “Fuzzy”. “Elestissearcc”: 3 edit distance from “Fuzzy”. After querying it, these queries produced a result: “Fyzzy” “Elastissearcc” The queries don’t: “tp” “Fyzyy” “Elestissearcc” Transpositions will allow your query to calculate the transpositions of two adjacent characters (ab -> ba) as 1 distance. transpositions For example, if we set the to true, we will get a result if we query with "leasticsearcc". transpositions But if we set it as , there will be no result from the Elasticsearch. false The Elasticsearch defaults the setting to true. transpositions We can’t set this setting to the Match Query. The Match Query will always calculate as 1 distance. transpositions Max Expansions will determine the maximum result you get from your query. max_expansions If you set the to 1 and there is 2 document in the Elasticsearch that are appropriate to your query, the Elasticsearch will only return 1. max_expansions Note that applies to shard level. So if you have many shards in the Elasticsearch, even if you set the to 1, the query might return more results. max_expansions max_expansion The default value for is 50. max_expansions Prefix Length is the number of prefix characters that is not considered in fuzzy query. prefix_length For example, if we set the to 1, we won't get any result if we query "llasticsearch". prefix_length The setting defaults to . prefix_length 0 Rewrite You can change parameter if you want to change the scoring of the results. rewrite You can find more information about the rewrite parameter in the . Elasticsearch documentation Conclusion Handling a typo in Elasticsearch is very easy and can improve the user’s experience. The simplest way to handle a typo is to just add in your Match Query. "fuzziness":"AUTO" If you want to tune the Query, there are some parameters that you can change with the "fuzziness" being the most important. Thank you for reading until the end! Previously published at https://codecurated.com/blog/how-to-handle-typos-in-elasticsearch-using-fuzzy-query/

Google

2022 - HackerNoon Contributor of the Year - Data Security

Visit my website!

Nominated for 2022 - HackerNoon Contributor of the Year - Data Security

Too Long; Didn't Read

Making security fun one episode at a time

How to Use Fuzzy Query Matches in Elasticsearch

Too Long; Didn't Read

Companies Mentioned

Brilian Firdaus

About Author

TOPICS

Languages

THIS ARTICLE WAS FEATURED IN...

How to Use Fuzzy Query Matches in Elasticsearch

Too Long; Didn't Read

Companies Mentioned

Brilian Firdaus

About Author

TOPICS

Languages

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES