Collaborative Judgements - Relevancy is a Team Sport

This tutorial assumes you have completed the previous tutorial, Tuning Relevance, and have set up a case with queries available. Here, we will guide you through setting up a collaborative environment to collect judgements from human raters.

Why Collaborative Judgements?

In Tuning Relevance, we created judgements in the role of relevance engineers: We followed our intuition to classify a result as relevant or irrelevant. This method works well for small-scale projects, like rating the top 10 results for a few queries. However, for larger projects, this approach can introduce biases and become inefficient.
By involving a team of human raters, we can scale up and improve the quality of judgements. Human raters evaluate query/document pairs by considering the information need of the query.

Information Needs

An Information Need is a conscious or unconscious desire for information. It helps raters evaluate whether a document satisfies the query's purpose.

For the star wars case, let’s define the information need for three queries:

  1. star wars: The user wants to find movies or shows from the Star Wars franchise. Potentially relevant are documentaries about Star Wars.
  2. star trek: The user wants to find movies or shows from the Star Trek franchise. Potentially relevant are documentaries about Star Trek.
  3. Toy Story: The user wants to find movies or shows from the Toy Story franchise. Potentially relevant are documentaries about Toy Story.

Adding Information Needs in Quepid

  1. Open a query (for example star wars) and select Toggle Notes
  2. Enter the information need in the first field and any additional notes in the second field: information_need.png
  3. Click Save

Note: It's easy to define information needs for three queries. You wouldn't want to go that way for dozens, hundreds or more queries. Fortunately, Quepid has you covered! On the case interface you can open a modal to upload queries with information needs by clicking Import and selecting Information Needs. import_information_needs.png

Create a Team

Teams in Quepid allow users to collaborate by sharing information and cases.

Creating a Team

  1. Go to Teams in the top-level navigation.
  2. Click + Add New. new_team.png
  3. Enter a team name (e.g., "star wars Human Raters") and click Create. name_new_team.png

Adding Team Members

  1. Select your team from the overview.
  2. Add members by typing their name or email in the input field. Every team member you want to add needs to have an active account in Quepid. add_a-team_member.png
  3. Click Add User to confirm.

Note: Don't worry if you don't have any team members to add. A Team can consist of one member only as well. That way, you can continue following the tutorial and add team members once they're available.

Share your Case with the Team

Sharing your case ensures all team members can access the queries, results, and judgements.

  1. In the Case Interface, select Share Case. share_case.png
  2. Choose the team to share with and confirm. share_case_modal.png

With that, every member of your Team has access to your case and can see the queries, the results, the available judgements, etc.

Create a Book of Judgements

A Book of Judgements in Quepid allows multiple raters to evaluate query/document pairs following best practices.

Creating a Book

  1. Go to Judgements in the case interface and click + Create a Book. create_a_book.png
  2. Configure the book:
    • Name: Use a descriptive name (e.g., Star Wars 0-3 Scale).
    • Teams: Share the book with your created team.
    • Scorer: Choose a scorer (e.g., DCG@10 for graded judgements).
    • Selection Strategy:
      • Single Rater: One judgement per query/document pair.
      • Multiple Raters: Up to three judgements per pair, enabling higher-quality assessments.
    • Other Settings: Leave default unless specific needs arise.
  3. Click Create Book.

new_book.png

This action leads you to the overview page of your book: book_overview.png

Populate the Book with Query/Document Pairs

  1. In the case interface, go to Judgements.
  2. Select the created book (e.g., Star Wars 0-3 Scale).
  3. Click Populate Book and confirm by selecting Refresh Query/Doc Pairs for Book. populate_book.png

This creates query/document pairs for the top 10 results of each query. For three queries, this generates 30 pairs.

Verify the Number of Query/Doc Pairs in the Book

  1. Navigate to the Book Overview via the Judgements tab.
  2. Confirm the book contains the expected number of query/document pairs (e.g., 30 for our example). book_overview_populated.png

At this point, the book will display "0 judgements" since no ratings have been added yet. The next tutorial, Quepid for Human Raters, will cover the judging process in detail.

By following these steps, you’ve set up a collaborative environment for human raters in Quepid. This method scales efficiently, reduces bias, and ensures high-quality judgements.

Happy judging!