How LLM as a Judge Works #

When you attach the AI Judge to a book in Quepid, it automatically sends evaluation requests to the OpenAI API using both your custom prompt and the query/document pair data.

Each request uses a prompt template that you configure in the AI Judge settings. This prompt is dynamically populated with the fields from each query/document pair, allowing the model to evaluate results in context.

Request Structure

Quepid uses OpenAI's chat completion API, which requires a messages array to simulate a conversation. For each document/query pair, the following structure is generated:

[
  {
    "role": "system",
    "content": "<your AI judge prompt>"
  },
  {
    "role": "user",
    "content": [
      {
        "type": "text",
        "text": "<textual query/document pair data>"
      }
    ]
  }
]

How Images are used in Evaluation #

If your document contains a field named image, or if any field is explicitly tagged with image:, Quepid handles it specially. The value of that field is treated as an image URL and is inserted into a separate section of the prompt to provide image context to the model.

This allows you to evaluate results based not only on textual relevance but also on visual content when applicable.

Request Structure

[
  {
    "role": "system",
    "content": "<your AI judge prompt>"
  },
  {
    "role": "user",
    "content": [
      {
        "type": "text",
        "text": "<textual query/document pair data>"
      },
      {
        "type": "image_url",
        "image_url": {
          "url": "<image url>"
        }
      }
    ]
  }
]