---
title: "Relevancy is a Team Sport"
url: "https://quepid-docs.dev.o19s.com/2/quepid/32/relevancy-is-a-team-sport"
---

# Collaborative Judgements - Relevancy is a Team Sport

This tutorial assumes you have completed the previous tutorial, **Tuning Relevance**, and have set up a case with queries available. Here, we will guide you through setting up a collaborative environment to collect judgements from human raters.

## Why Collaborative Judgements?

In **Tuning Relevance**, we created judgements in the role of relevance engineers: We followed our intuition to classify a result as *relevant* or *irrelevant*. This method works well for small-scale projects, like rating the top 10 results for a few queries. However, for larger projects, this approach can introduce biases and become inefficient.  
By involving a team of human raters, we can scale up and improve the quality of judgements. Human raters evaluate query/document pairs by considering the **information need** of the query.

## Information Needs

An [Information Need](https://en.wikipedia.org/wiki/Information_needs) is a conscious or unconscious desire for information. It helps raters evaluate whether a document satisfies the query's purpose.

For the _star wars_ case, let’s define the information need for three queries:

1. **star wars**: _The user wants to find movies or shows from the Star Wars franchise. Potentially relevant are documentaries about Star Wars._
2. **star trek**: _The user wants to find movies or shows from the Star Trek franchise. Potentially relevant are documentaries about Star Trek._
3. **Toy Story**: _The user wants to find movies or shows from the Toy Story franchise. Potentially relevant are documentaries about Toy Story._

## Adding Information Needs in Quepid

1. Open a query (for example _star wars_) and select _Toggle Notes_
2. Enter the information need in the first field and any additional notes in the second field:
![information_need.png](https://quepid-docs.dev.o19s.com/u/information_need-hxAllR.png) 
3. Click **Save**

> **Note:** It's easy to define information needs for three queries. You wouldn't want to go that way for dozens, hundreds or more queries. Fortunately, Quepid has you covered! On the case interface you can open a modal to upload queries with information needs by clicking **Import** and selecting **Information Needs**.
>  ![import_information_needs.png](https://quepid-docs.dev.o19s.com/u/import_information_needs-XnTn4W.png) 

## Create a Team

Teams in Quepid allow users to collaborate by sharing information and cases. 

### Creating a Team

1. Go to **Teams** in the top-level navigation.
2. Click **+ Add New**.
![new_team.png](https://quepid-docs.dev.o19s.com/u/new_team-As4QHZ.png) 
3. Enter a team name (e.g., "star wars Human Raters") and click **Create**.
 ![name_new_team.png](https://quepid-docs.dev.o19s.com/u/name_new_team-SJxPmd.png) 

### Adding Team Members

1. Select your team from the overview.
2. Add members by typing their name or email in the input field. Every team member you want to add needs to have an active account in Quepid.
 ![add_a-team_member.png](https://quepid-docs.dev.o19s.com/u/add_a-team_member-5z0ifh.png) 
3. Click **Add User** to confirm.

> **Note**: Don't worry if you don't have any team members to add. A Team can consist of one member only as well. That way, you can continue following the tutorial and add team members once they're available.

## Share your Case with the Team

Sharing your case ensures all team members can access the queries, results, and judgements.

1. In the **Case Interface**, select **Share Case**.
 ![share_case.png](https://quepid-docs.dev.o19s.com/u/share_case-PEbnTX.png) 
2. Choose the team to share with and confirm.
 ![share_case_modal.png](https://quepid-docs.dev.o19s.com/u/share_case_modal-ZekFl2.png) 

With that, every member of your Team has access to your case and can see the queries, the results, the available judgements, etc.

## Create a Book of Judgements

A **Book of Judgements** in Quepid allows multiple raters to evaluate query/document pairs following best practices.

### Creating a Book

1. Go to **Judgements** in the case interface and click **+ Create a Book**.
 ![create_a_book.png](https://quepid-docs.dev.o19s.com/u/create_a_book-lapx25.png) 
2. Configure the book:
    - **Name**: Use a descriptive name (e.g., _Star Wars 0-3 Scale_).
    - **Teams**: Share the book with your created team.
    - **Scorer**: Choose a scorer (e.g., _DCG@10_ for graded judgements).
    - **Selection Strategy**:
        - _Single Rater_: One judgement per query/document pair.
        - _Multiple Raters_: Up to three judgements per pair, enabling higher-quality assessments.
    - **Other Settings**: Leave default unless specific needs arise.
3. Click **Create Book**.

 ![new_book.png](https://quepid-docs.dev.o19s.com/u/new_book-007wHD.png) 

This action leads you to the overview page of your book:
 ![book_overview.png](https://quepid-docs.dev.o19s.com/u/book_overview-PgJkuv.png) 

## Populate the Book with Query/Document Pairs

1. In the case interface, go to **Judgements**.
2. Select the created book (e.g., _Star Wars 0-3 Scale_).
3. Click **Populate Book** and confirm by selecting **Refresh Query/Doc Pairs for Book**.
 ![populate_book.png](https://quepid-docs.dev.o19s.com/u/populate_book-obEQCx.png) 

This creates query/document pairs for the top 10 results of each query. For three queries, this generates 30 pairs.

### Verify the Number of Query/Doc Pairs in the Book

1. Navigate to the **Book Overview** via the Judgements tab.
2. Confirm the book contains the expected number of query/document pairs (e.g., 30 for our example).
 ![book_overview_populated.png](https://quepid-docs.dev.o19s.com/u/book_overview_populated-elSPFO.png) 

At this point, the book will display "0 judgements" since no ratings have been added yet. The next tutorial, **Quepid for Human Raters**, will cover the judging process in detail.

By following these steps, you’ve set up a collaborative environment for human raters in Quepid. This method scales efficiently, reduces bias, and ensures high-quality judgements.

Happy judging!