Invoke a Custom Evaluator
Runs an evaluator that already exists in the workspace. Accepts either a conversation or the structured input and output fields; when both are present the conversation wins.
Authorizations
Bearer authentication header of the form Bearer <token>, where <token> is your auth token.
Path Parameters
Accepts a bare id, id@version, or id@environment.
Body
Accepts two shapes. context names its fields after the template variables
they feed and is the one to use; the flat fields below are the legacy body,
folded into context when it is absent. Setting context wins.
The data to grade. When messages is present it is the conversation and
input.user_query is ignored; output.response is appended only when the
conversation carries no assistant turn. Mirrors graders-api buildGraderRequest.
Model to grade with, as a catalog id such as "openai/gpt-4o".
Only meaningful for a hub template of type llm_eval or ragas, which has no model of its own. A stored evaluator uses the model on its own definition and ignores this.
Latest user message. Folds into context.input.user_query.
The generated response from the model. Folds into
context.output.response.
The reference used to compare the output. Folds into
context.input.expected_output.
Knowledge base retrievals. Folds into context.input.retrievals.
The conversation that produced the output. Folds into
context.messages.
Template variables for evaluator prompt substitution. Folds into
context.variables.
Response
OK
Response wrappers keep each RPC's response type distinct, so a future field can be added to one without touching the other.
The verdict. Shaped to match WorkflowRunMinifiedEvalSchema, the body the TypeScript route returned, so existing consumers read the same JSON.