Improving User Experience with Search
Looking up the definition of User Experience (UX/UE) on any search platform returns a range of lofty explanations on how UX is part of every single aspect of a company. From the obvious ‘product’ to the shrink wrapping in which said product arrives, to the in-shop experience or the way the product is found. Just as we can’t not communicate, we can’t not provide a user experience; every detail matters.
Improving the experience as a whole can then become a herculean task. The good news is you don't need to tackle everything at the same time. One simple way to improve the experience your users have on your website is by improving something as trivial as your search bar.
It's easy to forget that although your website is one of your most powerful marketing channels, users rely on it as their main source of information. These two views can sometimes be conflicting. While the customer is on the webpage, you may be compelled to show them your newest product or how they can better utilise your services. Your customers, however, have their own goals and sometimes all they want to know is how to get in touch with you, or what the fuel consumption of that small car you manufacture is. Getting to that information can be a very efficient or a very frustrating experience for your customer.
"Our mission is to put our clients at the center of everything we do. Netcentric helped us deliver on our mission by creating a search experience based on how our users behave." Ines Anderegg, Chief Product Owner PostFinance
What happens
We've all been there. We go to a website in search of something particular – an article you want to show a friend or information you need to look up “quickly" – but end up leaving the page in frustration. It's sometimes easier to search using a search engine you trust for the specific words you’re looking for in combination with the site that contains that information, than to spend time looking for the information on the website itself. This can be due to a number of reasons.
The simple-minded query: It may be that a site only searches for OR combinations. This means if you search for "The man who sold the world", the search engine virtually fires 5 different queries, one for each unique word in your sentence and presents you with a set of results that may or may not include the David Bowie song you're looking for. If the search implementation was just a tiny bit smarter than that, it would at least filter out "the" and "who" words that occur in many of the documents inside your index.
The lack of help: Sometimes search engines are limited in their interpretation of human language. Maybe you’re only looking for one word, which you’re not 100% sure how to spell because it’s part of the song “Yellow Ledbetter” (and while Eddie Vedder may have one of the greatest voices in rock ‘n roll history, his enunciation leaves something to be desired). Or you might be searching for “Godless Killing Machine”, but turn up empty-handed because the authors are using the less apt term “bear” in their articles.
General disarray: It may also be difficult to find what you’re looking for if your search term is too common. Let’s say you’re searching through all 310 songs the Beatles released as a group for the term “love”. The probability of finding the song you’re looking for on the first page is small, as roughly 50% of the songs contain this word. Without weighting the results, there’s simply no telling if the song you’re looking for will appear on the first or last page.
What can you do
When we talk about search, like any other domain, we need to first establish a vocabulary.
Documents: Documents are pieces of content that you decide to provide to your search engine, which in turn allows the user to search through them. This could be a webpage, PDF, or any other piece of content that can be indexed.
Index: Indices are repositories in which your search engine will look for matches when a user launches a search query.
Indexing: A term that encompasses a wide range of activities, from the initial extraction and pre-processing of the document, to splitting the document into keywords, applying linguistic features, and storing it into an index structure.
Query: A list of parameters that are sent to your search engine, which returns search results. The actual query is not always 100% transparent to the user, since it may include language parameters or other filters that are implied but not actively selected.
Signals: Key-Value pairs that contain information about the search query which are not directly visible to your customers, but can contain information about the query – and the users themselves – in order to track their behavior and use the information to optimize the search results.
Phrase boosting
Let's start with optimising your query. As you may have gathered from the earlier references, simply using OR statements might not be enough to help users find what they are looking for. A great way to improve a query is to use phrases. With phrase boosting, you can check if the words in the query occur within a certain distance to each other. For the phrase "The Man Who Sold The World", the document for this song would be given more weight, as opposed to a document where the words "sold" and "world" occur several paragraphs apart.
Autocomplete
Autocomplete is another simple yet effective way to improve a user’s search experience. As a user, you can start typing "Cred...", and autocomplete will suggest ways to complete your search term – for example, “credit card” or “credible excuse for not doing homework”. Search engines offer multiple tools to provide autocomplete suggestions. One of them is to create an autocomplete dictionary with a simple collection of all the words in your index.
Now, a few words of caution: a dictionary is not an index. You wouldn’t be able to order autocomplete suggestions by occurrence, since a dictionary has no notion of this, while an index does. Another point to keep in mind is clean-up. If you’re selling cameras and unpublished an article from your website containing the words “shutter”, “aperture”, and “focal length”, you wouldn’t have to worry since those words probably occur on other pages. However, if you’ve just replaced your lineup and don’t want last year’s “Apollo 3310” showing up anywhere, you’d need to make sure to remove those unused terms from your dictionary.
It’s also possible to base your autocomplete suggestion on an index. This requires more effort, but also offers more features, like the aforementioned sorting by occurrence, or even a live search as the user is typing.
Did you mean?
I, the humble author of this article, have to make an effort to write “customer” and not “costumer”. Both words exist, so spell check won’t tell me I made a mistake. However, there’s a big difference in meaning (one being the delight of anyone who sells things, the other being the delight of someone in the market for a costume).
Normally, people won't notice typos like this since all the letters are there, and phonetically it sounds very similar. Computers are not so apt at overlooking tiny mistakes, though – so unless you have a typo in your content or you're dealing in costumes, this would lead to a "no results found" message on your search results page. ‘Did you mean’ lends your customer a virtual hand here. It not only searches for exact matches, but also for words that differ in a few letters from your search term. This lets you get away with searching for "Buitney" and your search engine will gently ask if you meant "Britney is finally free".
Synonyms
Synonyms are very powerful when ‘did you mean’ isn't good enough. Let's assume you're looking for "Saul Hudson" because you're a die hard fan of Guns N' Roses and a tiny bit snobbish. You obviously know that you're looking for the hatted guitar virtuoso, Slash, but how would a computer know that the two are the same person?
The answer is synonyms. This is a two-part job. You start off with an initial set of terms that you know will come up as synonyms because your clients might not be familiar with your branding, so you add the synonym "toaster" to your product called "bread make hot thingy" and everything else you think might warrant a synonym here. But afterwards, you need to monitor your customer's search terms closely. If you find that many people are looking for "Tafkap" but no results are found, you're either missing content, or you could simply add "Tafkap" to "Prince" since some relatively snobbish people are looking for the abbreviation of "The Artist Formerly Known As Prince" rather than typing "Prince" to get the same result.
Generic boostings and tunings
Vilfredo Pareto famously deduced that 80% of outcomes are due to 20% of causes. This is also true in search. Looking at your user’s most-used search terms, you'll quickly notice that there are maybe 10 terms that are often searched for, and the terms that come afterward rarely find their way into the search bar. Now let's say most users on your website were looking for "Account Fees" and that particular string occurs often, but the actual fees are only listed on a page called "Banking Packages". This can lead to a frustrating search experience for a large number of your customers. There are a few things you can do. You can boost terms that occur in important places like the page title. It doesn't help with your "Account Fees" problem, but it already improves the order of your search results, since people looking for "Banking Packages" will get that page first.
You can expand this kind of boosting to other attributes as well; maybe you want to prioritise pages over PDF documents or you wish to boost "how to" pages over "product descriptions" or vice versa. As long as you have a property that identifies the things you want to boost, you can do a lot. But you can also do too much and lose every hope of predicting any result set. For your "Account Fees" problem, you could also manually decide that the page "Banking Packages" should get a boost whenever someone is looking for "account fees". If you do this for your top search terms, your customers will have a very different experience. You can improve the life of 80% of your search users, with just 20% of the effort.
Automatic relevancy boosting
Relevance is the art of matching the user’s intent to content. In the end, we want to enable users to get the information they’re looking for as quickly as possible. Your search engine is capable of helping you with automatic relevancy boosting, but it needs information. By collecting signals from your users, something as simple as {"searchTerm=love, resultClicked="while my guitar gently weeps"}, you can find out what your users are looking for – and most importantly, what they click on. Armed with this information, your search engine can boost search results on its own.
This means that the order of the result set for the search term "love" is no longer random, nor manually boosted by page title or your intuition, but instead in the order of clicks each result gets. If you provide more information in the signals you send to your search engine, it can sort more elaborately. For instance, if you have private and institutional clients, they may be searching for very different things when they query "payment". If you add the segment to your signals, your search engine is capable of delivering tailored orders of the same result set and each of your customer segments will then see the most relevant results at the top of their search result page.
The best thing is that you can monitor the effects this boost has on the fly and see if there are things that have moved from the mysterious place called “Page 2” to the top of your first search result page. If more people start clicking on "Across The Universe" instead of "While My Guitar Gently Weeps" (simply because "Limitless undying love which shines around me like a million suns" is a very beautiful thing to hear), the order will change over time. Here again, the possibilities are endless. If your e-commerce platform runs on search, your search engine can predict what your users will search for next and show it in a convenient place. Just keep in mind that from a data protection point of view, it's better to feed your search engine with less data than to feed it with information that does not improve the search result. But if you have consent from your users, you can feed data points like location, device type, or previous action, and your search engine will match your content to your user’s intent perfectly.