How Many Queries and Documents Do I need? #

How many search results should be judged? #

The number of search results you should judge depends on a few factors and your organization's search team can provide guidance. As a general rule of thumb, 100 judgments (10 pages of 10 search results) is a good starting point and likely sufficient for most cases.

You may have noticed the frog icons? The backstory is that we needed an icon to call attention to some queries having unrated documents. Eric's son Asher loves frogs, and when Asher was asked to pick an icon, he picked the frog one! So, we talk about "chasing the frogs" to mean getting rid of unrated documents. The more frogs you have, the less confidence you have in accurate search metrics.

You can click the Frog Pond report link to get a sense of how many ratings you need.

Over time you can figure out how many ratings you need to do per week to stay on top of things.