AI Judges
AI Judges use large language models to grade search results, producing automated relevance judgments for an evaluation run. Each judge references an AI Provider and pairs it with a model identifier and a Handlebars prompt template.
A judge can be tested for connectivity and then used to start an AI judging run against a completed evaluation run.
List AI Judges
Lists all AI judges accessible to the current member. Admins see all judges; members see shared judges and their own.
Create an AI Judge
Creates a new AI judge. Admins can create shared judges (no member_id) or personal judges. Members can only create personal judges.
Delete AI Judges
Deletes the AI judges specified by the given unique identifiers.
Get an AI Judge
Gets the details of an AI judge.
Update an AI Judge
Updates an existing AI judge. Only the owner or an admin can update.
Test an AI Judge
Tests connectivity to an AI judge by sending a simple prompt. If the judge is configured
Start AI Judging
Starts an AI judging run on a completed evaluation run.
List AI Judging Runs
Lists all AI judging runs for an evaluation run.
Cancel AI Judging Run
Cancels a running AI judging run.