Invoke a Custom Evaluator
Runs an evaluator that already exists in the workspace. Accepts either a conversation or the structured input and output fields; when both are present the conversation wins.
Authorizations
Bearer authentication header of the form Bearer <token>, where <token> is your auth token.
Path Parameters
Accepts a bare id, id@version, or id@environment.
Body
Accepts two shapes. context names its fields after the template variables
they feed and is the one to use; the flat fields below are folded into
context when it is absent. Setting context wins.
The data to grade. When messages is present it is the conversation and
input.user_query is ignored; output.response is appended only when the
conversation carries no assistant turn.
Model to grade with, as a catalog id such as "openai/gpt-4o".
Only meaningful for a hub template of type llm_eval or ragas, which has no model of its own. A stored evaluator uses the model on its own definition and ignores this.
Latest user message. Folds into context.input.user_query.
The generated response from the model. Folds into
context.output.response.
The reference used to compare the output. Folds into
context.input.expected_output.
Knowledge base retrievals. Folds into context.input.retrievals.
The conversation that produced the output. Folds into
context.messages.
Template variables for evaluator prompt substitution. Folds into
context.variables.
Response
OK
The verdict. Its shape is fixed so existing consumers read the same JSON.
Discriminator for the verdict shape: "string", "number", "boolean", "string_array", "rouge_n", "bert_score", "llm_evaluator", "http_eval".
The verdict. Dynamic by design — a boolean pass, a numeric score, a categorical label, a list of labels and the nested rouge_n object all arrive here.
Trace reference of the evaluator's own span. Optional so an absent reference is omitted rather than emitted as an empty string.
How the run ended, as distinct from passed: "passed", "condition_failed",
"failed" or "timed_out". A string, not an enum, because the engine owns the
vocabulary.
The guardrail's decision when the evaluator has one, the grader's own
judgement otherwise. Always present, so read guardrail_config to detect
a guardrail, not this.
Set by the classifying graders (moderation, PII, secret detection), which report which categories tripped rather than a single verdict.
Optional so an absent score is omitted rather than reported as 0.0, which a consumer would read as maximum uncertainty.