We Built A Search Engine With MeiliSearch and JavaScript: Here's How You Can Too

Struggle to find the chemistry Nobel prize winner for 1979? Try out MeiliSearch's super-fast-search JavaScript tutorial to find Nobel prize winners in a matter of milliseconds. 🌬️ is a client for MeiliSearch written in JavaScript. MeiliSearch is a powerful, fast, open-source, easy to use and deploy search engine. Both searching and indexing are highly customizable. Features such as typo-tolerance, filters, and synonyms are provided out-of-t he-box. MeiliSearch JS A working example can be found on codeSandBox and in the meilisearch-vue directory. The goal of this project is to teach you how to: Create a new index Modify settings for your index Add data to your index Ignore stop words Enable facet filters for faster data filtering We’ll be using both the API directly as the JavaScript wrapper to show you how both work. Throughout this tutorial, we’ll be using a cool Nobel prize winners data set to show you a couple of examples. First, let’s take a look at the requirements. Requirements Here are the requirements to be able to follow this tutorial: Node.js installation Running an (via Docker or a ) Don’t want to set up your own MeiliSearch instance, try out our to create a temporary index for 72h. instance of MeiliSearch DigitalOcean droplet MeiliSearch sandbox for sending requests from the terminal (Postman works as well) cURL MeiliSearch-js wrapper and dependencies (see ). installation guide Let's get started with the creation of your first index. Project setup and MeiliSearch-js installation To follow along, we need to set up our JavaScript project and install MeiliSearch-js. Create a new folder and run the following command in your terminal. npm init -y This will prepare your project setup. Next, we can add the MeiliSearch-js dependency. npm install meilisearch Lastly, let’s create a file called index.js in your project. We’ll use this file to add our JavaScript code. touch index.js Good, let’s get started! Step 1: Create your first index First of all, let’s start with the preparation. We assume here that you have a and can access this via your localhost or a public IP address. running instance of MeiliSearch To keep things simple, we aren’t using a master key. A master key allows you to protect all API endpoints for your Meilisearch instance. We highly recommend when using Meilisearch in production or hosting it via a publicly accessible IP address - e.g. DigitalOcean droplet. Important: setting a master key To verify if you can reach your MeiliSearch instance, try querying for the available indexes. If you haven’t created any indexes, you should see an empty array as a result. Below you find the cURL command you can execute from your terminal. curl http://127.0.0.1:7700/indexes Now, let’s edit our file to create a connection object. We’ll use this connection object to create our first index. First, we need to import the MeiliSearch dependency. Furthermore, the property accepts the IP address of your MeiliSearch instance. index.js host MeiliSearch = ( ) main = () => { client = MeiliSearch({ : }) indexes = client.listIndexes() .log(indexes) } main() const require 'meilisearch' const async const new host 'http://127.0.0.1:7700' const await console Note that we’ve added some extra code that uses the object to query for all indexes and then prints the result. client To execute the file, you can run it with the command. node node index.js Finally, let’s create our first index. As we are working with Nobel prizes, let’s name the index . We can use the function to create a new index. To verify the successful creation of our index, let’s query again for all indexes to see the newly created index. prizes createIndex MeiliSearch = ( ) main = () => { client = MeiliSearch({ : }) indexes = client.listIndexes() .log(indexes) client.createIndex( ) updatedIndexes = client.listIndexes() .log(updatedIndexes) } main() const require 'meilisearch' const async const new host 'http://127.0.0.1:7700' const await console await 'prizes' const await console You should see the following result being printed to your terminal. [ { : , : , : , : , : } ] name 'prizes' uid 'prizes' createdAt '2020-10-21T16:08:23.052004747Z' updatedAt '2020-10-21T16:08:23.052024846Z' primaryKey null The property is optional. By default, Meilisearch sets the property equal to the property. However, when dealing with a lot of similar indexes, it’s useful to define a human-readable name that differs from the property. Note: name name uid uid Also, Meilisearch hasn’t set a primary key yet for the prizes index. When we add data in the next step, the as our data set contains an id field. primary key will be automatically picked Index created? Good! Let’s explore the Nobel prizes data set. Step 2: Adding the Nobel prizes dataset First of all, let’s explore the dataset briefly. The original dataset we used for this example comes from , however, we’ve modified the dataset slightly to fit our use case. nobelprize.org You can explore the . The structure of the data looks like this. modified dataset here Each object contains an ID that will serve as the primary key. MeiliSearch will automatically search for a property that contains the word , such as or . If you want to use a dataset that doesn’t contain an , you can still manually . id prizeId objectId id set a primary key Further, we find properties such as year, category, firstname, surname, motivation, and share. { : , : , : , : , : , : , : } id "991" year "2020" category "chemistry" firstname "Emmanuelle" surname "Charpentier" motivation "for the development of a method for genome editing" share "2" Now, let’s download the dataset as a JSON file using cURL. We are using the property to define an output file for the downloaded contents. -o curl -L https://gist.githubusercontent.com/michielmulders/5328ffcabd04bdcaded389a278728304/raw/3c172f0bbc248f99096c2e34653432b983704f07/prizes.json -o prizes.json Next, we need to add the dataset to the MeiliSearch instance. Let’s upload the dataset to the index. Note that the URL changed slightly as we are adding to the index: . Make sure the filename for the property matches the filename of your prizes JSON file. prizes documents prizes indexes/prizes/documents --data curl -i -X POST \ --header \ --data @prizes.json 'http://127.0.0.1:7700/indexes/prizes/documents' 'content-type: application/json' To verify if the data has been uploaded successfully, let’s query for all documents. You should see all Nobel prize objects. curl http://127.0.0.1:7700/indexes/prizes/documents Success! Next, let’s use some code to add an extra document to our index. prizes Step 2.1: Add documents using the MeiliSearch-js client We’ve just added documents using our terminal. It’s time to add an extra document using JS code. Let’s define an array that contains new documents we want to add to our index. Note, we first need to retrieve our index so we can use this index object to add documents. prizes MeiliSearch = ( ) main = () => { client = MeiliSearch({ : }) index = client.getIndex( ) documents = [ { : , : , : , : , : , : , : } ] response = index.addDocuments(documents) .log(response) } main() const require 'meilisearch' const async const new host 'http://127.0.0.1:7700' const 'prizes' const id '12345' year '2021' category 'chemistry' firstname 'Your' surname 'Name' motivation 'for the development of a new method' share '1' let await console // => { "updateId": 0 } When you add a new document, Meilisearch returns an object containing an . Using the you can track the document addition process until it is processed or failed. updateId update methods In step 3, let’s learn how to search for documents. Step 3: Search for Nobel prize documents Searching for documents is pretty simple. Again, we first need to retrieve the index object. Next, we can use the index object to search for a particular query. As an example, we are looking for to show MeiliSearch’s type-tolerance. chemisytr MeiliSearch = ( ) main = () => { client = MeiliSearch({ : }) index = client.getIndex( ) search = index.search( ) .log(search) } main() const require 'meilisearch' const async const new host 'http://127.0.0.1:7700' const 'prizes' const await 'chemisytr' console This returns a huge list of results. Let’s learn how you can add filters to, for example, limit the number of results. Change the following line to add an object that accepts filters. search = index.search( , { : }) const await 'chemisytr' limit 1 This returns the following result. { : [ { : , : , : , : , : , : , : } ], : , : , : , : , : , : } hits id '991' year '2020' category 'chemistry' firstname 'Emmanuelle' surname 'Charpentier' motivation '"for the development of a method for genome editing"' share '2' offset 0 limit 1 nbHits 111 exhaustiveNbHits false processingTimeMs 1 query 'chemisytr' Next, we want to modify the settings for the prizes index to eliminate stop words. Step 4: Modify index settings to eliminate stop words Now, let’s take a look at the settings for our prizes index. You can access the settings via the exposed API like so: http://localhost:7700/indexes/prizes/settings You’ll see the following result with an empty array. stopWords { : [ , , , , , ], : , : [ ], : [ ], : [ ], : { }, : [ ] } rankingRules "typo" "words" "proximity" "attribute" "wordsPosition" "exactness" distinctAttribute null searchableAttributes "*" displayedAttributes "*" stopWords synonyms attributesForFaceting We can achieve the same using JavaScript code like so. index = client.getIndex( ) settings = index.getSettings() .log(settings) const 'prizes' const await console Now, let’s add a couple of we want to eliminate. Stop words are frequently occurring words that don’t have any search value. stop words For example, no products exist that are called or . To improve the search speed, we want to avoid searching for such stop words. When the user looks for the search query “a mask”, the MeiliSearch engine will automatically remove the part and look for the word . a the a mask In this example, we want to eliminate the following stop words: an the a First, let’s check how many results we receive when querying for the word . the index = client.getIndex( ) results = index.search( ) .log(results.nbHits) const 'prizes' const await 'the' console The above query for returns 494 hits. Now, let’s modify our script to eliminate the above stop words. the index.js index = client.getIndex( ) response = index.updateSettings({ : [ , , ] }) .log(response) const 'prizes' const await stopWords 'a' 'an' 'the' console To verify the effectiveness of our settings change, let’s query again for the word . Now, this query should return results. Cool right? the 0 You might have forgotten a particular stop word such as . If you send a new request to your API, this will overwrite the old configuration. Therefore, make sure to send the full list of stop words every time you want to make changes. Quick tip: and updateSettings Let’s move on! Step 5: Define facet filters is a feature provided out-of-the-box by MeiliSearch. Faceting allows classifying search results into categories that are called facets. This allows the MeiliSearch engine to search through data much faster using facet filters. Faceted search Facet filters are most useful for numbers or . For example, Nobel prizes are awarded for a fixed list of categories. This makes up a great facet filter. The same is true for the year property. enums You can only use facet filters for strings or arrays of strings. In our example, the property is stored as a string which allows us to filter by the property. Note: year year Below you find an example snippet that adds facet filters for the properties and . You can always verify which facet filters have been added by consulting the settings for your index. year category index = client.getIndex( ) response = index.updateSettings({ : [ , ] }) const 'prizes' const await attributesForFaceting 'category' 'year' Step 5.1: Experimenting with facet filters Now, I want to query all Nobel prize winners with the name Paul. This returns 14 results. index = client.getIndex( ) search1 = index.search( ) .log( ) const 'prizes' const await 'paul' console `Search 1 hits: ` ${search1.nbHits} // 14 Following, I want to filter down the results by the category chemistry. Note that we send an extra data property with the request that allows us to set facet filters. This property accepts an array of strings. Each string contains two words, separated by a colon. The first part is the facet and the second part contains the value. That’s how we get the following facet filter for . chemistry: category:Chemistry index = client.getIndex( ) search2 = index.search( , { : [ ]}) .log( ) const 'prizes' const await 'paul' facetFilters 'category:Chemistry' console `Search 2 hits: ` ${search2.nbHits} // 5 This query returns five results. Lastly, I want to add some extra facet filter to filter by both category and year. I want to return Nobel prize winners for the year 1995, 1996, or 1997. Luckily, MeiliSearch allows for combining multiple facet filters. If you want to add multiple values for the same facet filter, you’ll have to use another array inside the facet filters array. Look at the example query below. index = client.getIndex( ) search3 = index.search( , { : [ , [ , , ] ]}) .log( ) const 'prizes' const await 'paul' facetFilters 'category:Chemistry' 'year:1995' 'year:1996' 'year:1997' console `Search 3 hits: ` ${search3.nbHits} // 2 Ultimately, this returns only two results that match our needs. { :[ { : , : , : , : , : , : , : },{ : , : , : , : , : , : , : } ], : , : , : , : , : , : } hits id '287' year '1997' category 'chemistry' firstname 'Paul D.' surname 'Boyer' motivation '"for their elucidation of the enzymatic mechanism underlying the synthesis of adenosine triphosphate (ATP)"' share '4' id '281' year '1995' category 'chemistry' firstname 'Paul J.' surname 'Crutzen' motivation '"for their work in atmospheric chemistry, particularly concerning the formation and decomposition of ozone"' share '3' offset 0 limit 20 nbHits 2 exhaustiveNbHits false processingTimeMs 0 query 'paul' Nice! Lastly, let’s play with ranking rules for our MeiliSearch engine. Step 6: Play with MeiliSearch ranking rules In step 3, we’ve shown you how MeiliSearch handles typos by querying for instead of . chemisytr chemistry However, you might have noticed that the settings for your index list many different ranking rules. are what defines relevancy in MeiliSearch. They affect how a result is considered better than another. Ranking rules are sorted top-down by order of importance. Ranking rules We can change the order of these ranking rules based on your use case to fine tune the speed of your MeiliSearch instance. It’s even possible to remove certain ranking rules. { : [ , , , , , ], : , : [ ], : [ ], : [ , , ], : { }, : [ , ] } rankingRules "typo" "words" "proximity" "attribute" "wordsPosition" "exactness" distinctAttribute null searchableAttributes "*" displayedAttributes "*" stopWords "a" "an" "the" synonyms attributesForFaceting "year" "category" Let’s query for again with the above standard ranking rules. We are most interested in results 10 to 14 in the returned list. To get this list, we use the method for arrays. Paul slice index = client.getIndex( ) search = index.search( ) .log(search.hits.slice( , )) const 'prizes' const await 'Paul' console 10 14 We get the following result. Note that results with id and are listed first. 737 637 [ { : , : , : , : , : , : , : }, { : , : , : , : , : , : , : }, { : , : , : , : , : , : , : }, { : , : , : , : , : , : , : } ] id '737' year '2001' category 'medicine' firstname 'Sir Paul' surname 'Nurse' motivation '"for their discoveries of key regulators of the cell cycle"' share '3' id '637' year '1964' category 'literature' firstname 'Jean-Paul' surname 'Sartre' motivation '"for his work which, rich in ideas and filled with the spirit of freedom and the quest for truth, has exerted a far-reaching influence on our age"' share '1' id '217' year '1954' category 'chemistry' firstname 'Linus' surname 'Pauling' motivation '"for his research into the nature of the chemical bond and its application to the elucidation of the structure of complex substances"' share '1' id '50' year '1945' category 'physics' firstname 'Wolfgang' surname 'Pauli' motivation '"for the discovery of the Exclusion Principle, also called the Pauli Principle"' share '1' Now, let’s change the ranking rules where we prefer the rule to be the most important rule. From the documentation, we can find the following information about the . wordsPosition wordsPosition rule “Results are sorted by the position of the query words in the attributes: find documents that contain query terms earlier in their attributes first.” For our example, we want to prioritize names that start with the word over composed names such as or . Let’s change the order first using our code example. Paul Sir Paul Jean-Paul index = client.getIndex( ) index.updateSettings({ : [ , , , , , ] }) const 'prizes' await rankingRules "wordsPosition" "typo" "words" "proximity" "attribute" "exactness" Now, let’s query for Paul again. Notice how the position for both and has changed. Sir Paul Jean-Paul [ { : , : , : , : , : , : , : }, { : , : , : , : , : , : , : }, { : , : , : , : , : , : , : }, { : , : , : , : , : , : , : } ] id '217' year '1954' category 'chemistry' firstname 'Linus' surname 'Pauling' motivation '"for his research into the nature of the chemical bond and its application to the elucidation of the structure of complex substances"' share '1' id '50' year '1945' category 'physics' firstname 'Wolfgang' surname 'Pauli' motivation '"for the discovery of the Exclusion Principle, also called the Pauli Principle"' share '1' id '737' year '2001' category 'medicine' firstname 'Sir Paul' surname 'Nurse' motivation '"for their discoveries of key regulators of the cell cycle"' share '3' id '637' year '1964' category 'literature' firstname 'Jean-Paul' surname 'Sartre' motivation '"for his work which, rich in ideas and filled with the spirit of freedom and the quest for truth, has exerted a far-reaching influence on our age"' share '1' You can find more examples for each ranking rule in the . MeiliSearch documentation Step 6.1: Define your own ranking rule Lastly, it’s possible to define your own ranking rule. That’s pretty exciting right? You should know that custom ranking rules only work for numeric values. However, it doesn’t matter that you’ve defined a numeric value as a string such as our property. Now, let’s get funky and remove all ranking rules and add a custom ranking rule for the property. year year You can define an ascending or descending sorting rule. index = client.getIndex( ) index.updateSettings({ : [ ] }) const 'prizes' await rankingRules "desc(year)" Next, let’s search for again. Now, notice that the results are sorted by the property as expected. Paul year [ { : , : , : , : , : , : , : }, { : , : , : , : , : , : , : }, { : , : , : , : , : , : , : }, … ] id '995' year '2020' category 'economics' firstname 'Paul' surname 'Milgrom' motivation '"for improvements to auction theory and inventions of new auction formats"' share '2' id '834' year '2008' category 'economics' firstname 'Paul' surname 'Krugman' motivation '"for his analysis of trade patterns and location of economic activity"' share '1' id '764' year '2003' category 'medicine' firstname 'Paul C.' surname 'Lauterbur' motivation '"for their discoveries concerning magnetic resonance imaging"' share '2' That’s it! Conclusion That’s it for this MeiliSearch and JS tutorial. This tutorial has taught you how to use the MeiliSearch API, create indexes, modify index settings, and define facet filters for more accurate and faster searching. For more information, make sure to take a look at the and the . documentation JS API wrapper on GitHub Liked using MeiliSearch, make sure to show us some love by on GitHub! giving MeiliSearch a star Previously published at https://blog.meilisearch.com/how-to-search-nobel-prize-winners-faster-with-meilisearch-api-for-javascript/