---
title: "Tuning Relevance"
url: "https://quepid-docs.dev.o19s.com/2/quepid/31/tuning-relevance"
---

# Tuning Relevance

This tutorial assumes you have have finished the Setting Up Your First Case tutorial.  It is split into two sections: Creating Judgements and then Tuning Relevance.

## Creating Judgments
In order to do relevancy tuning, you must have some judgements available to you to evaluate your search quality against.

> While you _can_ judge documents right in a Case, in general it's better to use a Book to manage the judgements and let Subject Matter Experts or Expert Users do the judging work!

### Making Binary Judgements

Let's start with the _star wars_ query by clicking it. The list of movies matching the star wars query will be shown. To the left of each movie search result is a gray box. We use this box provide a relevance judgment for each movie. Click the _Select Scorer_ link and click _P@10_. This allows us to make a binary judgment on each search result and label each as either relevant (1) or not relevant (0). Now, for each search result, look at the fields returned (that we selected when we created the case) and click the _gray box_. If you think this search result is relevant to the query click **Relevant**. Otherwise, click **Irrelevant**

 ![rating-binary.png](https://quepid-docs.dev.o19s.com/u/rating-binary-iWrXKs.png) .

Congratulations! You have just provided a human judgment to a search result!  

### Graded Judgements for Nuance
As you probably know, search relevancy is often not a binary decision. Sometimes one result is more relevant than another result. We want to see the movie _Star Wars_ returned above _Plastic Galaxy: The Story of Star Wars Toys_. Both movies are relevant, but the former is more relevant. 

We'll now change the **Scorer** we are using to one that supports Graded Judgements.  Click the _Select Scorer_ link and select the _DCG@10_ option and click the _Select Scorer_ button. With this scorer selected for our **Case** we can judge our search results on a scale of `0` (poor) to `3` (perfect).

 ![rating-graded.png](https://quepid-docs.dev.o19s.com/u/rating-graded-111j3Q.png) 

> Don't worry about what the scorers actually mean - that is out of scope for us as a human rater. We will leave the scorers to the search relevance engineers!

Look at the title and overview of the first search result. Is this movie relevant for the query _star wars_? Click the gray box by the movie and make your selection. Is it a poor match, a perfect match, or somewhere in between? There is likely no correct answer - two people may give different answers and that's ok. What's important is that you take your time and provide your judgments consistently across queries and search results.

#### Who is keeping score?
As you assign judgments, you will likely notice the score at the top left of the page changes. It's important to remember you are not trying to get a high score, a low score, or any particular score. For our efforts as a rater we are not at all concerned about a score. The search relevance engineers will know what to do with the score! Just remember, it is not your goal or responsibility to have any influence on the score. 

_The score is not a reflection of your efforts._


## Tuning Relevance

Now let's do some search tuning! Our _star wars_ query returns movies that are at least somewhat related to the Star Wars movies but we are probably wanting to see the actual Star Wars movies first instead of movies such as 'The Stars of Star Wars' and 'Plastic Galaxy: The Story of Star Wars Toys.' So let's do some tuning to see if we can improve our search results. We are using the AP@10 scorer. With our human judgments, our score for the current query is 0.21. Click the _Tune Relevance_ link to open the side window and the Query tab.

 ![tune-relevance-query.png](https://quepid-docs.dev.o19s.com/u/tune-relevance-query-KpzJEC.png) 

Let's use our search engineering knowledge to improve our query's score! Improving our search is heavily an experimental and iterative process. We may try some things that work and some things that don't work and that's ok! We can learn from what doesn't work in our iterations. To start, let's change our search to use phrase fields `pf` to boost results where the search query (star wars) appears in the title. We'll also use `edismax`. Now, our query looks like:

`q=#$query##&pf=title^20&defType=edismax`

Click the _Rerun My Searches_ button to run our new query. Let's look at our search results now. The first result is the movie Star Wars so that's great and an obvious improvement! The AP@10 score has increased to 0.41. The second movie is 'Star Wars Spoofs' and 'Battle Star Wars.' Are those movies what our searchers want to see? Probably not. The other Star Wars movies would be better results.

 ![tune-relevance-new-query.png](https://quepid-docs.dev.o19s.com/u/tune-relevance-new-query-Hwgmgo.png) 

Let's iterate on our query and see if we can improve some more. What should we try? Maybe there are other fields in our schema that we could be using. The TMDB database has a`genres` field that contains a list of genres for each movie. We can try restricting our search results to just action movies using the `genres` field. This will help remove some of the irrelevant movies from the search results.

`q=#$query## AND genres:"Action"&defType=edismax`

This provides a huge gain in our score although we might have a few frogs indicating there are new results that have not been judged. Will we always know the user is searching for action movies? Maybe, maybe not - it depends on your use-case. This is just an example of how we can use Quepid to iterate on our queries.

#### What do the frog icons mean?
The little frog icon tells you that you have some documents for a query 
unrated, and how many of them.  The more unrated documents, the less 
confident you can be as a Relevancy Engineer that you have accurate metrics.


### Understanding How the Query was Processed

It can be helpful to understand how the original user query was processed. Clicking the _Explain Query_ button will open a window that has some panes to help you understand what happened to the query.  The first pane is the **Params** listing.  It is showing exactly what parameters were sent to the search engine for the query, and can help you understand what other influences are made on the query.

 ![tune-relevance-explain.png](https://quepid-docs.dev.o19s.com/u/tune-relevance-explain-Kb8KqL.png) 

The second panel, **Explain Query Parsing** shows how the original user query was parsed into the final query.  Especially for multi word queries you can see how they are expanded into a much longer query, and can help explain why some seemingly random documents might get match!

 ![explain-query-parsing.png](https://quepid-docs.dev.o19s.com/u/explain-query-parsing-63wGZ9.png) 

### Viewing the Query Tuning History

As we iterate, it can be helpful to see what we have previously tried. The **History** tab will show the history of our previous queries.

 ![tune-relevance-history.png](https://quepid-docs.dev.o19s.com/u/tune-relevance-history-bVHKrz.png) 

Since every **Try** is related to it's parent **Try**, you can visualize them in a tree format.

Under the _History_ tab at the top you will see a _Visualize your tries_ link. Clicking this link opens a new window that shows a visual representation of your query history. Hover over each node to see the query that was run. Clicking on a node in the visualization will re-run that query. 

The history link visualization provides a powerful way to examine your previous queries. As part of Quepid's goal to reduce experiment iteration time, the history link visualization can help you reduce the iteration time by showing a comprehensive look at your past iterations and helping you to not inadvertently duplicate any previous iterations.

Here is an example of a straight line set of **Tries** that eventually branched:

 ![history-link.png](https://quepid-docs.dev.o19s.com/u/history-link-NAlrIU.png)

## Measuring your Overall Progress

**Snapshots** are how Quepid lets you take a point in time and make it comparable to support seeing how things are changing. Is your search relevancy improving, is it declining? Snapshots provide you with that knowledge.

After changes to the query have been made and you are at a point where you would like to record your progress, click the _Create snapshot_ link.  Give the **Snapshot** a name like _Baseline_.

 ![snapshot-create.png](https://quepid-docs.dev.o19s.com/u/snapshot-create-UeL5TL.png) 

Having snapshots available as you work will allow you to compare your progress over time.

To compare snapshots, click the _Compare snapshots_ link. 

 ![snapshot-compare.png](https://quepid-docs.dev.o19s.com/u/snapshot-compare-9io9aI.png) 

In the window that is shown, select the snapshot you want to compare against and click the _Update diff _settings button. Quepid will update the search results to show both sets to show the current search results against the previous snapshot. 

 ![snapshot-compare-view.png](https://quepid-docs.dev.o19s.com/u/snapshot-compare-view-6n7YF7.png) 

You can see that our current **Try 3** results is on the left, and the previously created **Snapshot Baseline** is on the right.  The overall **Case** score bumped up  by 2 points from .95 to .97 as measured by nDCG@10.

You can see that this was due to a five point improvement in the _star wars_ query.  Opening the query, we can see that "perfect" movie _Star Wars_ moved to the top in Try 3, and pushed down the "good" movie _Star Wars: The Clone Wars_ from the first position.

 ![snapshot_row1.png](https://quepid-docs.dev.o19s.com/u/snapshot_row1-jAokQ4.png) 

To stop comparing snapshots, click the _Compare snapshots_ link and select the _Disabled_ option and click the _Update diff_ settings button.

Snapshot comparison is a powerful tool for quickly visualizing the difference in your queries and will assist in your future search tuning efforts. The comparison of the search metrics between these two snapshots provides feedback to the search team! The snapshots can be used to measure progress over time as your search team works to improve your organization's search relevance.