Invoke a Custom Evaluator
Runs an evaluator that already exists in the workspace. Accepts either a conversation or the structured input and output fields; when both are present the conversation wins.
timeout.call_timeout to control the timeout in milliseconds, up to 600000. The default is 120000 milliseconds unless configured otherwise. This uses the same request shape as the Responses API. Synchronous calls execute directly in graders-api and stop when the request deadline expires. Runtime failures return an error response; a valid negative verdict returns passed: false.
For LLM-as-a-judge Evaluators (llm_eval), set timeout.call_timeout to limit each provider attempt in milliseconds. Set retry.count to the number of retries after the initial attempt, from 0 to 5, and retry.on_codes to the HTTP status codes that allow a retry. Omitting on_codes uses the gateway’s default transient codes. Omitting retry preserves the existing retry policy. For LLM judges the timeout applies to each provider attempt; for other Evaluators it applies to execution. Caller cancellation still stops synchronous work, and the server caps each invocation at ten minutes including retries.
timeout.call_timeout is set, the gateway also includes HTTP 408 among the retryable codes, matching the Responses API.Authorizations
Bearer authentication header of the form Bearer <token>, where <token> is your auth token.
Path Parameters
Accepts a bare id, id@version, or id@environment.
Body
Accepts two shapes. context names its fields after the template variables
they feed and is the one to use; the flat fields below are folded into
context when it is absent. Setting context wins.
The data to grade. When messages is present it is the conversation and
input.user_query is ignored; output.response is appended only when the
conversation carries no assistant turn.
Model to grade with, as a catalog id such as "openai/gpt-4o".
Only meaningful for a hub template of type llm_eval or ragas, which has no model of its own. A stored evaluator uses the model on its own definition and ignores this.
Latest user message. Folds into context.input.user_query.
The generated response from the model. Folds into
context.output.response.
The reference used to compare the output. Folds into
context.input.expected_output.
Knowledge base retrievals. Folds into context.input.retrievals.
The conversation that produced the output. Folds into
context.messages.
Template variables for evaluator prompt substitution. Folds into
context.variables.
Response
OK
The verdict. Its shape is fixed so existing consumers read the same JSON.
Discriminator for the verdict shape: "string", "number", "boolean", "string_array", "rouge_n", "bert_score", "llm_evaluator", "http_eval".
The verdict. Dynamic by design — a boolean pass, a numeric score, a categorical label, a list of labels and the nested rouge_n object all arrive here.
Trace reference of the evaluator's own span. Optional so an absent reference is omitted rather than emitted as an empty string.
How the run ended, as distinct from passed: "passed", "condition_failed",
"failed" or "timed_out". A string, not an enum, because the engine owns the
vocabulary.
The guardrail's decision when the evaluator has one, the grader's own
judgement otherwise. Always present, so read guardrail_config to detect
a guardrail, not this.
Set by the classifying graders (moderation, PII, secret detection), which report which categories tripped rather than a single verdict.
Optional so an absent score is omitted rather than reported as 0.0, which a consumer would read as maximum uncertainty.