Open How to Use the Docs

Welcome to Quepid!

You are on your way to better search quality.

Ready to get deeper with Quepid? There are many ways of building your Quepid "qompetency", so the documentation is built around trying to meet you where you are.

The docs are organized in four quadrants using the Diátaxis framework:

Each Quadrant in listed as a separate chapter. We start with the Tutorials and How-To Guides first, and then have the deeper Explanation and Reference sections.

Many of these pages will link out to other locations that have more details.

Want to contribute to the docs? Ping Eric at epugh@opensourceconnections.com for a invite to become a writer.

Open What is Quepid?

What is Quepid?

Before we go furthur, what exactly is Quepid? Quepid is a tool to help you improve your organization's search results. It is a web-based application and can either be used at https://app.quepid.com or it can be installed on your own hardware. Quepid makes improving your app's search results a repeatable, reliable engineering process that the whole team can understand

Did you know?

Did you know? Quepid is an open source project open to everyone! If you have a passion for helping others improve search relevance visit the project on GitHub to learn how to contribute.

Quepid was created and designed to address three pain-points often experienced during search relevancy improvement:

• Collaboration - Our collaboration stinks! Making holistic progress on search requires deep, cross-functional collaboration. Sending emails between teams or tracking search requirements in spreadsheets won't provide the improvem

Open Structure of Tutorials

Structure of Tutorials

These tutorials are set up to guide you through learning Quepid. Pick the ones that map to your specific role.

The tutorials assume you are using the hosted version of Quepid at http://app.quepid.com.

I am a Relevance Engineer

• Setting Up Your First Case (has scoring)

• Tuning Relevance (and digging into query explanation)

• Relevancy is a Team Sport

• Working With the Judges

I am a Judge who needs to Rate Documents

• Judging Documents

I am an Ops Person Setting up Quepid

• Installing Quepid

• Being Able to Develop Quepid

Open Setting Up Your First Case

Setting Up Your First Case

The first time you sign up on Quepid you will be prompted with a big blue button to Create Your First Relevancy Case.

A Case refers to all of the queries and relevance tuning settings for a single search engine. For example, if we are running a movie rental store and we want to improve our "Star Wars" search results, we may create a new case and call it "star wars." Under this case we will perform our searches and provide our judgments.

Creating a New Case

Click the Create Your First Relevancy Case. If you don't see it, then click the Relevancy Cases menu at the top and click Create a case.

The first piece of information we are asked to provide is a name for the Case. Enter "star wars" and click Continue.

Now, we need to set up the connection to the Search Endpoint. We're going to use a publicly hosted Solr search engine as our Search Endpoint. Click t

Open Tuning Relevance

Tuning Relevance

This tutorial assumes you have have finished the Setting Up Your First Case tutorial. It is split into two sections: Creating Judgements and then Tuning Relevance.

Creating Judgments

In order to do relevancy tuning, you must have some judgements available to you to evaluate your search quality against.

While you can judge documents right in a Case, in general it's better to use a Book to manage the judgements and let Subject Matter Experts or Expert Users do the judging work!

Making Binary Judgements

Let's start with the star wars query by clicking it. The list of movies matching the star wars query will be shown. To the left of each movie search result is a gray box. We use this box provide a relevance judgment for each movie. Click the Select Scorer link and click P@10. This allows us to make a binary judgment on each search result and label each as either relevant (1) or not relevant (0). Now, for each search result, look at the fields returned (that we

Open Relevancy is a Team Sport

Collaborative Judgements - Relevancy is a Team Sport

This tutorial assumes you have completed the previous tutorial, Tuning Relevance, and have set up a case with queries available. Here, we will guide you through setting up a collaborative environment to collect judgements from human raters.

Why Collaborative Judgements?

In Tuning Relevance, we created judgements in the role of relevance engineers: We followed our intuition to classify a result as relevant or irrelevant. This method works well for small-scale projects, like rating the top 10 results for a few queries. However, for larger projects, this approach can introduce biases and become inefficient.
By involving a team of human raters, we can scale up and improve the quality of judgements. Human raters evaluate query/document pairs by considering the information need of the query.

Information Needs

An Information Need is a conscious or unconscious desire for in

Open Quepid for Human Raters

This tutorial builds on the previous one, Collaborative Judgements, and assumes you have already set up a book with query/document pairs ready for evaluation. In this guide, we will explore the Human Rating Interface, which is designed to streamline the process of assigning judgements to query/document pairs.

The role of a human rater is accessible to anyone with the right task description. No technical background is required—just the ability to assess how well a document matches the information need for a given query.

Overview of the Human Rating Interface

The Human Rating Interface is thoughtfully structured to assist raters in providing accurate and consistent judgements. The interface consists of the following key components:

1. Query: Displays the query associated with the query/document pair being evaluated.
2. Information Need: Highlights the

Open How to Deploy Quepid on Digital Ocean

Quepid can be deployed on your own hardware using Docker. The Installation Guide will walk you through the deployment and creating users.

This page needs updating.

Make sure it's a howto and not a reference

Open How to Add a Recency Scorer

How to Add a Recency Scorer

A Recency Scorer lets you understand the average age of the content you are returning.

1. Go to the Scorers page and choose Add New to create the new Scorer.
2. Give it a meaningful name like Recency or Age of Product that works for you.
   
3. For the actual JavaScript code, start with this basic scorer:

const k = 10; // @Rank
let rank = 0;
let score = 0;
baseDate = new Date("2023-08-15").getTime();
eachDoc(function(doc, i) {
  docDate = doc['publish_date'];
  const diffTime = (baseDate - new Date(docDate).getTime());
  const diff = Math.ceil(diffTime / (1000 * 60 * 60 * 24));
  score = score + diff;
  rank = rank + 1
}, k);
score = rank > 0 ? score / rank : 0.0;
setScore(score);

1. We need to specify what field in the docs has the date field. Change the publish_date to your date field.
2. In order to calculate a recency score, we have to decide what is out of date, and that is the baseDate. In this example, we ha

Open How to Create a Personal Access Token for API access

How to Create a Personal Access Token for API Access

Quepid has a rich set of APIs that let you manipulate all the data in Quepid. To ensure that when you access the APIs you are following the correct permissions structure you have to set up a Personal Access Token.

1. In the top right click on your user icon and from the drop down menu choose My Profile.
2. Scroll down to the Personal Access Tokens section.
   
3. Click Generate New Token to create a Personal Access Token.
4. You will see an example similar to curl -H "Authorization: Bearer 9557fac2245f8b5a52897e2ee9dca010dd8f4288c678adb15d43682ef12a4a0f" https://app.quepid.com/api/users/20 of how to use the generated token in Curl.
   

The Personal Access Token is sometimes referred to as a Bearer Token.

Open How to Include Images

How to Include Images

Image formatting rules are specified via the Field Specification defined for a Case.

Assuming you have the full path to the image stored in your search index, then you can specify in your field spec that you either want a thumbnail size image via thumb:product_image_url or you can get a larger image via image:product_image_url.

Now, if you have relative URL's defined such as poster_path then you can specify in the Field Specification a JSON formatted specification:

{
  "name": "poster_path", 
  "type":"thumb", 
  "prefix": "https://www.example.org/images/thumbs"
}

You can specify the type as either thumb or image.

Multiple Images

Quepid can only show a thumbnail and a regular size image, you can't have multiple images or multiple thumbnails. So in that case, index one image as your "hero" image and reference that field.

Images in Books

When you populate a book, Quepid consults the Field Specification to figure out how

Open How to Connect Quepid to Elastic Cloud

Prerequisites

1. You have an active Elastic Cloud account
2. You have a deployment with a searchable index

This how to describes the three necessary processes to connect Quepid to your Elastic Cloud deployments:

1. Configure Elastic Cloud deployment to accept browser requests from Quepid
2. Create an API Key in Elastic Cloud

3. Connect Quepid to Elastic Cloud with the appropriate settings

   Configure your Elastic Cloud deployment to accept browser requests from Quepid

4. Log into your Elastic Cloud account at https://cloud.elastic.co/home.

5. Select Manage next to the Deployment you want to connect Quepid to.

6. Under your deployment name select Edit.

7. Select Manage user settings and extensions. ![manage_EC_deployment_settings.png](https://quepid-docs.dev.o19s.com/u/manage_ec

Open Core Concepts

Core Concepts

First, let's walk through a few of the core concepts in Quepid.

Book: A book refers to a set of queries and documents and their ratings independent of the search engine. A single book can provide the source information for multiple cases.

Case: A case refers to all of the queries and relevance tuning settings for a single search engine. If you want to work with multiple instances of Solr or Elasticsearch (or any other search engine or API), you must create a separate case for each one.

Query: Within a case, queries are the keywords or other search criteria and their corresponding set of results that will be rated to determine the overall score of a case.

Rating: Ratings are the numerical values given to a result that indicates how relevant a particular search result is for the query. How each rating is interpreted depends on the scorer used for the query (or case), but usually the higher the number the more relevant the result is.

Result: Within a query

Open Quepid App Settings

There are many application level settings for Quepid that you can use to optimize YOUR deployment of Quepid. These are detailed here: https://github.com/o19s/quepid/blob/main/docs/operating_documentation.md

Hey Eric: You need to reorg that page! It has both operating docs, but also some how to type content.

Open Quick Start Wizard

Quick Start Wizard

The "Quick Start Wizard", which launches on first login or any time you create a new case, will guide you through configuring your Solr, Elasticsearch, or other search endpoint and setting up your case.

What you need to complete the wizard

• A Solr or Elasticsearch instance that is accessible from your browser
• A list of desired fields from your search engine

*Note: You can use the demo Solr or Elasticsearch instances provided to test Quepid

Naming Your Case

First you will be required to enter a case name. Select something descriptive to distinguish between your cases.

Connecting your Search Engine

Quepid does not require any installation on your server. All you need to do is indicate which search engine you are working with, and provide the URL to your search engine. A wide variety of search engines are supported.

You will be able to update this URL at any point in the s

Open Case Screen

Case Screen

Now that we have the basic setup out of the way, let's take a look at the major elements of Quepid's Case screen.

1. Relevancy Case Dropdown - Select or create cases to work on.

2. Case Information - Displays case summary including name, average query score, and current iteration in dashboard history.

3. Case Actions - Shows available actions including scorer selection, snapshot management, sharing options, and relevancy tuning panel toggle.

4. Queries -Add new queries and view existing ones. Click arrow to view and rate individual query results.

5. Relevancy Tuning Panel - Modify search relevancy settings and tune searches. Panel is closed by default and accessible via case actions.

6. Quepid Support - Access support chat window in bottom right corner for assistance from search relevancy experts.

Open Managing your Query Set

Manage Your Queries

Adding Queries

Adding additional queries is simple. Just type in your search term(s) and click Add query and Quepid will populate the results. A newly added query will display an empty score, the title, and the number of results found. Click on the arrow to expand a single result:

Want to add a number of queries all at once? Just separate them with a ;, like so: star wars;star trek and click Add query.

Rating Queries

Queries are rated through the dropdown to the left of each result. Clicking on the dropdown will bring up the rating scale based on the scorer you have selected.

If you haven't picked a specific scorer, then the Quepid system default scorer AP@10 will be used. It has a binary Irrelevant or Relevant choice.

Open Relevance Panel

Relevance Panel

To modify your search settings, open the relevancy tuning panel by clicking the tune relevance link in the case actions area.

The default panel allows you to modify the Solr or Elasticsearch query parameters directly. Update your query here and click Rerun my searches to see the results update.

Magic Variables

"Magic variables" are placeholders. You can add a list of queries that Quepid will automatically run against your search engine. While you can specify which parameters are sent with the call, you need to know how to connect your query list to these parameters. This is where "magic variables" come in.

There are 3 types of "magic variables":

Query Magic Variable

The #$query## string represents the query variable. Quepid replaces this pattern with your full query. This is the simplest and most important "magic variable."

Example: Let's say you have:

1. A movies index using S

Open Importing and Exporting Ratings

Importing ratings will override your existing ratings. Proceed with caution!

Ratings can be imported into existing queries and cases. You may want to import ratings if you have previously rated search results in another application other than Quepid. Ratings can be imported from CSV, Rated Ranking Evaluator, or in the common Learning To Rank (LTR) format. To import ratings, click the Import link. A window will be shown to begin the import.

First, importing ratings can clear your existing queries. If all of your work has been done outside of Quepid and you are now bringing your work into Quepid you can select the Clear existing queries option to remove all queries from Quepid prior to the import. Please be sure you want to do this because all queries in Quepid will be lost.

Next, select the format of your ratings to import. Choose either CSV, Rated Ranking Evaluator, or

Open Managing Snapshots

Managing Snapshots

A important step in the benchmarking process for Quepid is taking regular snapshots of your case over time and comparing them to ensure that relevancy is improving.

Creating Snapshots

When you are ready to take a snapshot, click on Create Snapshot in the case actions area to bring up the snapshot dialog box. From here, give your snapshot a descriptive name and click Take Snapshot. The current ratings for all of your queries are now saved.

Comparing Snapshots

Once you have made improvements to your search relevancy settings, you will want to judge your improvement. Click on_"Compare snapshots_ to view your current results with a previous snapshot. You can now see the scores and results side-by-side for comparison.

Open Managing Cases

Managing Cases

Also on the case dashboard, you can manage existing cases. You can see both the cases you own and those that have been shared with you.

For each set of cases, you can filter the list by name and number of documents per page, and paginate through pages if needed. You can click on any case name to open it in the main Quepid dashboard.

Each case can be archived or renamed. Cases should be archived when no longer active, and you can access archived cases using the View archived cases link at the bottom of the page.

Open Importing and Exporting Cases

Open Manage Your Team

Manage Your Team

Teams in Quepid allow you to group your users based on their roles and cases. For example, you can create teams for your raters and your search relevance engineers as shown in the screenshot below. Case can be assigned to teams to provide access to your company's search team members. You can also make custom scorers available to a team. Using Quepid's Teams helps keep your search relevancy efforts organized. To access the Teams, click the Teams menu at the top of any page.

To create a team, click the Add New button. You will be asked to give the new team a name. All team names must be unique in each deployment of Quepid.

Creating a Team

To add users to a team, click the name of the desired team to open the team detail page. On this page, enter the user's email address or display name and click the Add User button.

![team4.p

Open Managing Scorers

Managing Scorers

Scorers are run over all your queries to calculate how good the search results are according to the ratings you've made.

You can specify whatever scale you want, from binary (0 or 1) to graded (0 to 3 or 1 to 10) scales. Each scorer behaves differently depending on whether it handles graded versus binary scales.

Classical Scorers Shipped with Quepid

These are scorers that come with Quepid. k refers to the depth of search results evaluated. Learn more in our "Choosing Your Search Relevance Metric" blog post.

Scorer	Scale	Description
Precision P@10	binary (0 or 1)	Precision measures the relevance of the entire results set. It is the fraction of the documents retrieved that are relevant to the user's information need. Precision is scoped to measure only the top k results.
Average Precision AP@10	binary (0 or 1)	Average Preci

Open Custom Scorers

Custom Scorers

Quepid comes with classic Information Retrieval Scorers, that we refer to as Communal scorers. Sometimes however you want your own Custom Scorer.

To access the scorers click the Scorers link at the top of any page.

On the scorers page, click the Communal tab to show only the scorers that are available to all users of Quepid. Click the Custom tab to show only scorers that you have created and shared with a team. To create a new scorer, click the _Add New _button. The New Scorer window will be shown.

First, give your scorer a name. Next, you can enter the JavaScript code for your scorer. Refer to Creating a Custom Scorer to learn more about the code requirements. Next, choose the scale for the scorer. Finally, you can optionally provide labels for the scale by clicking the Show Scale Labels option. When fini

Open API Documentation

Curious about the API's that Quepid exposes? Check out the documentation at https://app.quepid.com/apipie.

Note: The generated docs are NOT comprehensive, and you may need to consult the source code or open up your web inspector to see the HTTP traffic to learn about other APIs.

Quepid's APIs enable you to automate processes and integrate Quepid with other products you or your company uses. The APIs are REST-based. Responses are JSON, and errors are reported via standard HTTP codes in addition to JSON-formatted error information in the HTTP response bodies of relevant requests. We use Personal Access Token for authentication and authorization.

Open How Many Queries and Documents Do I need?

How Many Queries and Documents Do I need?

How many search results should be judged?

The number of search results you should judge depends on a few factors and your organization's search team can provide guidance. As a general rule of thumb, 100 judgments (10 pages of 10 search results) is a good starting point and likely sufficient for most cases.

You may have noticed the frog icons? The backstory is that we needed an icon to call attention to some queries having unrated documents. Eric's son Asher loves frogs, and when Asher was asked to pick an icon, he picked the frog one! So, we talk about "chasing the frogs" to mean getting rid of unrated documents. The more frogs you have, the less confidence you have in accurate search metrics.

You can click the Frog Pond report link to get a sense of how many ratings you need.

Over time you can figure out how many ratings you need to do per week

Open Using Notebooks for Analytics

Using Notebooks for Analytics

Data scientists are gonna data science... - Eric

At some point, you decide you want to dig into the data collected by Quepid to do more advanced analytics. Maybe you are curious about Inter Rater Reliablity? Or you want to do some deeper comparison between Cases or Snapshots such as Jaccard comparison? This is what the Notebooks interface is for.

Quepid ships with some example Jupyter notebooks that are meant as starting points for analyzing data. Each notebook is runnable, they all leverage a public dataset to demonstrate how they work. You can just click the double arrow play button to rerun them. Note, the first time you run the notebooks it takes a bit to load dependencies, and then they run quickly.

To use them, right click on the example notebook and copy it to your own file that is outside the /examples directory. Otherwise you may lose your customization

Open Thinking about Team and Sub Teams