Skip to main content

Judgment Lists

A Judgment List is a pre-existing set of relevance judgments that you upload to Releval. Unlike inline judgments that are made interactively during an evaluation run, judgment lists come from external sources such as:

  • Click-through logs or analytics
  • Previous evaluation campaigns
  • Crowdsourced relevance assessments
  • Expert annotations

File format​

A judgment list is uploaded as a file with one judgment per row. The file extension selects the parser: JSON Lines (.jsonl / .json / .ndjson), CSV (.csv / .tsv), and Parquet (.parquet) are all supported.

Each row describes a single relevance judgment: a query, the document being judged, and the grade. The grade is an integer that must fall within the range of the scale you choose on upload — binary, graded, or detailed. Optionally, a row can also carry a snapshot of the document itself, which is useful when the originating system may not return the same document text later.

JSON Lines example​

{ "query_id": "q1", "query": "running shoes", "doc_id": "sku-1001", "grade": 4 }
{ "query_id": "q1", "query": "running shoes", "doc_id": "sku-1002", "grade": 2 }
{ "query_id": "q2", "query": "winter coat", "doc_id": "sku-2050", "grade": 3 }

CSV example​

query_id,query,doc_id,grade
q1,running shoes,sku-1001,4
q1,running shoes,sku-1002,2
q2,winter coat,sku-2050,3

Uploading a Judgment List​

A judgment list belongs to a corpus, so create the corpus first if it doesn't already exist. On upload you also pick the type of the judgments (whether they are explicit ratings, implicit signals derived from behaviour, golden examples, or crowdsourced) and the scale the grades use.

In the UI​

  1. Navigate to Judgment Lists and click Upload.
  2. Select the file to upload.
  3. Enter a Name, choose the Corpus it belongs to, the Type, and the Scale the grades use.
  4. Click Upload.

Using the API​

curl -X POST "https://${RELEVAL_HOST}/api/v1/judgment-lists/upload" \
-H "Authorization: Bearer ${TOKEN}" \
-F 'name=My Judgment List' \
-F 'corpus_id=${CORPUS_ID}' \
-F 'type=explicit' \
-F 'scale=graded' \
-F 'judgments=@judgments.jsonl'

Managing Judgment Lists​

List All Judgment Lists​

curl "https://${RELEVAL_HOST}/api/v1/judgment-lists" \
-H "Authorization: Bearer ${TOKEN}"

You can filter by endpoint type:

curl "https://${RELEVAL_HOST}/api/v1/judgment-lists?endpoint_type=elasticsearch" \
-H "Authorization: Bearer ${TOKEN}"

Delete Judgment Lists​

curl -X DELETE "https://${RELEVAL_HOST}/api/v1/judgment-lists?judgment_list_id=${ID}" \
-H "Authorization: Bearer ${TOKEN}"

Training Data Generation​

Judgment lists can be used to generate training data for learning-to-rank models. This feature is available for Elasticsearch and OpenSearch endpoints.

curl -X POST "https://${RELEVAL_HOST}/api/v1/features/training" \
-H 'Content-Type: application/json' \
-H "Authorization: Bearer ${TOKEN}" \
--data @- <<EOF
{
  "judgment_list_id": "${JUDGMENT_LIST_ID}",
  "index": "products",
  "query_template_ids": ["${TEMPLATE_ID_1}", "${TEMPLATE_ID_2}"]
}
EOF

This executes each query from the judgment list against the endpoint using the specified templates, extracting features for each judged document. The resulting training data pairs features with relevance grades, suitable for training ranking models.