How Judgements are Averaged into a Rating in a Case
When combining the judgments from multiple judges, we can make a variety of assumptions. If we assume:
Judges are generally subject matter experts (SMEs)
Judges are rational
Judges have individual biases when interpreting relevance
Appealing to 1, we take the three highest out of all the judgments there are. Note, many additional judgments could all agree with these first three. Here we are being optimistic. If the judges do not agree, we appeal to numbers 2 and 3. If a rational judge deems the candidate to be poorer than the other judge or judges, we will trust them, and use their value. Here we are both being pessimistic, concerning the individual judgment, and optimistic, concerning the direction of the bias. That is, we expect judges to overrate, in the general case.
If we only have two judges, then we just average the two judgements. And if we only have one judge, then that is the rating.
As cases grow, and the relative quality of the judges becomes clearer, alternative combinations could be used.