Managing Scorers

Scorers are run over all your queries to calculate how good the search results are according to the ratings you've made.

You can specify whatever scale you want, from binary (0 or 1) to graded (0 to 3 or 1 to 10) scales. Each scorer behaves differently depending on whether it handles graded versus binary scales.

Classical Scorers Shipped with Quepid

These are scorers that come with Quepid. k refers to the depth of search results evaluated. Learn more in our "Choosing Your Search Relevance Metric" blog post.

Scorer	Scale	Description
Precision `P@10`	binary (0 or 1)	Precision measures the relevance of the entire results set. It is the fraction of the documents retrieved that are relevant to the user's information need. Precision is scoped to measure only the top k results.
Average Precision `AP@10`	binary (0 or 1)	Average Precision measures the relevance to a user scanning results sequentially. It is similar to precision, but weights the ranking so that a 0 in rank 1 punishes the score more that a 0 in rank 5.
Reciprocal Rank `RR@10`	binary (0 or 1)	Reciprocal Rank measures how close to the number one position the first relevant document appears. It is a useful metric for known item search, such as a part number. A relevant document at position 1 scores 1, at positon 2 scores 1/2, and so forth.
Cumulative Gain `CG@10`	graded (0 to 3)	Information gain from a results set. It just totals up the grades for the top k results, regardless of ranking.
Discount Cumulative Gain `DCG@10`	graded (0 to 3)	Builds on CG, however it includes positional weighting. A 0 in Rank 1 punishes your score significantly more than Rank 5.
normalized Discount Cumulative Gain `nDCG@10`	graded (0 to 3)	nDCG takes DCG and then measures it against a ideal relevance ranking yielding a score between 0 and 1. Learn more about nDCG (including some gotchas) on the wiki.

Selecting a Scorer

By default, several scorers are available. From the main dashboard, click Select scorer to choose from the available options. If you want to use a different rating scale, click Create New Scorer to access the custom scorers page.

Creating a Custom Scorer

This page lets you change which scorer is your default scorer when you create new Cases.

From the custom scorers page, click New Scorer. From here, you can name your new scorer, provide custom scoring logic in JavaScript, and select the scoring range.

What can my code do?

Your code validates the search results that came back. Below is an API available for you to work with:

Function	Description
`docAt(i)`	The document at the i'th location in the displayed search results from the search engine, including all fields displayed on Quepid. Empty object returned on no results.
`docExistsAt(i)`	Whether the i'th location has a document.
`eachDoc(function(doc, i) {}, num)`	Loop over docs. For each doc, call passed in function with a document, doc, and an index, i. You can pass in an optional `num` parameter to specify how many of the docs you want to use in the scoring (eg. to only use the top 5 results, pass in the number `5` as a second parameter. Default: `10`
`eachRatedDoc(function(doc, i) {}, num)`	Loop over rated documents only. Same arguments as eachDoc. Note: Needs rated documents loaded before usage, see `refreshRatedDocs`
`refreshRatedDocs(k)`	Refresh rated documents up to count `k`. This method returns a promise and must be run before using `eachRatedDoc`. You should call `then()` on the promise with a function that kicks off scoring.
`numFound()`	Solr has found this many results.
`numReturned()`	The total number of search results here.
`hasDocRating(i)`	True if a Quepid rating has been applied to this document.
`docRating(i)`	A document's rating for this query. This rating is relative to the scale you have chosen for your custom scorer.
`avgRating(num)`	The average rating of the returned documents. This rating is relative to the scale you have chosen for your custom scorer. You can pass in an optional `num` parameter to specify how many of the docs you want to use in the scoring (eg. to only use the top 5 results, pass in the number `5` as a second parameter. Default: `10`
`avgRating100(num)`	The average rating of the returned documents. This rating is on a scale of 100 (i.e. the average score as a percentage). You can pass in an optional `num` parameter to specify how many of the docs you want to use in the scoring (eg. to only use the top 5 results, pass in the number `5` as a second parameter. Default: `10`
`editDistanceFromBest(num)`	An edit distance from the best rated search results. You can pass in an optional `num` parameter to specify how many of the docs you want to use in the scoring (eg. to only use the top 5 results, pass in the number `5` as a second parameter. Default: `10`
`setScore(number)`	Sets the query's score to number and immediately exits
`pass()`	Pass the test (score it 100), immediately exits
`fail()`	Fail the test (score it 0), immediately exits
`assert(condition)`	Continues if the condition is true, otherwise immediately fails the test and exits
`setScore(number)`	Sets the query's score to number and immediately exits