> ## Documentation Index > Fetch the complete documentation index at: https://docs.orq.ai/llms.txt > Use this file to discover all available pages before exploring further. > Run an evaluator against a specific input and output pair. Returns a score and optional reasoning based on the evaluator's configured criteria. # Invoke a Custom Evaluator ## OpenAPI ````yaml post /v2/evaluators/{id}/invoke openapi: 3.1.0 info: title: orq.ai API version: '2.0' description: orq.ai API documentation servers: - url: https://api.orq.ai security: - ApiKey: [] tags: - name: Guardrail Rules - name: Policies - name: Routing Rules - name: Files description: File upload and retrieval operations. - name: FilesService - name: Projects description: Projects organize resources within a workspace - name: ProjectsService - name: Skills description: >- Skills are modular instructions you can use to codify processes and conventions - name: SkillsService - name: Responses - description: >- Run agents on a cadence — cron, interval, or one-off. Minimum firing interval is 1 hour. name: Agent Schedules - name: Reporting description: >- GenAI reporting API over canonical analytics rollups. Accepts a metric name, time range, grain, group-by, and filters; returns a typed time series and optional totals. - name: ReportingService description: |- ReportingService exposes a single QueryReport RPC that maps allowlisted analytics payloads onto safe rollup queries. Callers never send SQL; the backend picks the rollup family and grain from the metric catalogue, the requested range, and the requested grouping. externalDocs: url: https://docs.orq.ai description: orq.ai Documentation paths: /v2/evaluators/{id}/invoke: post: tags: - Evals summary: Invoke a Custom Evaluator operationId: InvokeEval parameters: - schema: type: string description: Evaluator ID required: true description: Evaluator ID name: id in: path requestBody: content: application/json: schema: type: object properties: query: type: string description: Latest user message output: type: string description: The generated response from the model reference: type: string description: The reference used to compare the output retrievals: type: array items: type: string description: Knowledge base retrievals messages: type: array items: type: object properties: role: type: string enum: - system - developer - assistant - user - exception - tool - prompt - correction - expected_output description: The role of the prompt message content: anyOf: - type: string - type: array items: oneOf: - type: object properties: type: type: string enum: - text text: type: string required: - type - text description: Text content part of a prompt message - type: object properties: type: type: string enum: - image_url image_url: type: object properties: id: type: string format: ulid pattern: ^[0-9A-HJKMNP-TV-Z]{26}$ readOnly: true description: The orq.ai id of the image url: type: string description: >- Either a URL of the image or the base64 encoded data URI. examples: - https://picsum.photos/id/1/200/300 - >- data:image/jpeg;base64,/9j/4QDcRXhpZgAASUkqAAgAAAAGABIBAwABAAAAAQAAABoBBQABAAAAVgAAABsBBQABAAAAXgAAACgBAwABAAAAAgAAABMCAwABAAAAAQAAAGmHBAABAAAAZgAAAAAAAABIAAAAAQAAAEgAAAABAAAABwAAkAcABAAAADAyMTABkQcABAAAAAECAwCGkgcAFAAAAMAAAAAAoAcABAAAADAxMDABoAMAAQAAAP//AAACoAQAAQAAADIAAAADoAQAAQAAADIAAAAAAAAAQVNDSUkAAABQaWNzdW0gSUQ6IDH/2wBDAAgGBgcGBQgHBwcJCQgKDBQNDAsLDBkSEw8UHRofHh0aHBwgJC4nICIsIxwcKDcpLDAxNDQ0Hyc5PTgyPC4zNDL/2wBDAQkJCQwLDBgNDRgyIRwhMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjL/wgARCAAyADIDASIAAhEBAxEB/8QAGgAAAwEBAQEAAAAAAAAAAAAAAAUGBAMCAf/EABgBAAMBAQAAAAAAAAAAAAAAAAACAwEE/9oADAMBAAIQAxAAAAGJ8VqTNw8a3ojTukdLSJLgZZlZdd6TV3CHYj9EWPM6bhUaNGKXfClLIL11F7/cfh02i8CpWBzWy5A6I+PgAAB//8QAIhAAAQQCAgIDAQAAAAAAAAAAAQACAwQSEwURFDIgISIj/9oACAEBAAEFAnQPYtaw6HSrfpk1Zpi6U1uqwOvALjX+aJeJ6RjdEYR/LS1X2a7q4WGdtkqbVqFlkbfMcrVEWn16sDBsmY+e+GQvsPmOYWSF47i5rI4ZjhycvcuZWX1sTao2TGJsBsvCL8iSs12nk4D2se/w/8QAHhEAAgIBBQEAAAAAAAAAAAAAAAECESEQEhMgMUH/2gAIAQMBAT8BErRHKo4is0RwSk2b5CT+lrWXnT//xAAbEQACAgMBAAAAAAAAAAAAAAAAAQIREiExIP/aAAgBAgEBPwEbpj1szFtWTVkYJGMRyiuFMssh3x//xAAqEAABAwIEBQMFAAAAAAAAAAABAAIRAxIQITFREyAiI2EyQXEEQlJigf/aAAgBAQAGPwLqGGYOeDmbqWDMYFjiah2aF2aFNnk9RVVv1HcjMSrqOmzlDha7D0hVNjnhfw3WFuqzU1Yt+FbTogjdxXoCFVz7Q0Q5TRYC78no3hwj3jpVwDS8+VLzOIMRC4tIxP27fHhERcVaCMtYHJ+moVmU+wC1IKnm/nL/AP/EACIQAAICAgEEAwEAAAAAAAAAAAERACExQVEQYXGBobHB0f/aAAgBAQABPyEgQjvByg05hUo2LiFjFYI7W43EDiVaHswr6T+yviNWy8EGiuNR40cj6MPOzDuOGmTHfziwMq+76EShwYJ8LmPkvUA0DK7y8RUIwU4Wf1moJxadQT0/YfMYyGLPkJiCZ6Rw+gHEOyLDRUgnkkfo7QoEZmabf4SsxCxwB3gBm9QgAYBbhGy6dySFOwYVeIcGHyxFUUo4LJ6JIxoBUkQiSuY4Ooc9f//aAAwDAQACAAMAAAAQK/6xMJVNTaRAi9cA/8QAHBEBAQEAAgMBAAAAAAAAAAAAAQARECExQVFh/9oACAEDAQE/ENJdCAjLX2X0Wzsr5v0mqx7MJOb3z//EAB0RAQEBAAAHAAAAAAAAAAAAAAEAERAgITFhcaH/2gAIAQIBAT8QxhwZqxessJqwhet4oA+Y0FeQTtx//8QAIxABAAICAQUAAgMAAAAAAAAAAQARITFBUWFxgZEQobHB0f/aAAgBAQABPxB2tTC0xLVUH7haUN2g1vP+RNGh8gq4UTGuGBUZ4MHv3iLSr8RIRqqi6Yv1carl4onc0eoWxjg8RD1aVFzk6f6b9/Y70zAti+Kx71GvjJeYuVsc6QwgBr5D/a/xfW1rFYoOcjjrK7p8CjoUUGXu5eMk0ZEmr6iVhzq4Hhj0LyLoGUvPiDGiXFOoUx6C6YKtA/EOTpu4FoAQGnL1SFVoWNDsQt27RtZg8GOQ6mm5/hiDjAW62BbV23sLxFbn20oMclu8B8lPIPaU4y+1hNDrowVllngQ2LIsGVq3iXxdrsPZu+8NsVhdHaO+tLVdwQ3CKME80BZTdNRSUt7WbPXP6zm5YlVv5//Z detail: type: string description: >- Specifies the detail level of the image. Currently only supported with OpenAI models required: - url required: - type - image_url description: >- The image part of the prompt message. Only supported with vision models. - type: object properties: type: type: string enum: - file description: >- The type of the content part. Always `file`. file: type: object properties: file_data: type: string description: >- The file data as a data URI string in the format 'data:;base64,'. Example: 'data:image/png;base64,iVBORw0KGgoAAAANS...' uri: type: string description: >- URL to the file. Only supported by Anthropic Claude models for PDF files. mimeType: type: string description: >- MIME type of the file (e.g., application/pdf, image/png) filename: type: string description: >- The name of the file, used when passing the file to the model as a string. required: - type - file - type: 'null' description: >- The contents of the user message. Either the text content of the message or an array of content parts with a defined type, each can be of type `text` or `image_url` when passing in images. You can pass multiple images by adding multiple `image_url` content parts. Can be null for tool messages in certain scenarios. tool_calls: type: array items: type: object properties: id: type: string index: type: number type: type: string enum: - function function: type: object properties: name: type: string arguments: type: string description: JSON string arguments for the functions required: - name - arguments required: - type - function tool_call_id: type: - string - 'null' required: - role - content description: >- The messages used to generate the output, without the last user message model: type: string description: Model to use for LLM-based evaluators (e.g. "openai/gpt-4o") responses: '200': description: Returns the result of the evaluator run content: application/json: schema: oneOf: - type: object properties: type: type: string enum: - string original_value: type: - string - 'null' value: type: - string - 'null' required: - type title: String - type: object properties: type: type: string enum: - number original_value: type: - number - 'null' value: type: - number - 'null' format_options: anyOf: - type: object properties: significant_digits: type: number required: - significant_digits - type: object properties: style: type: string enum: - currency currency: type: string enum: - USD required: - style - currency required: - type - value title: Number - type: object properties: type: type: string enum: - boolean value: anyOf: - type: boolean - type: string - type: number - type: 'null' required: - type - value title: Boolean - type: object properties: type: type: string enum: - string_array values: type: array items: type: - string - 'null' required: - type - values title: String Array - type: object properties: type: type: string enum: - rouge_n value: type: object properties: rouge_1: type: object properties: f1: type: number precision: type: number recall: type: number required: - f1 - precision - recall rouge_2: type: object properties: f1: type: number precision: type: number recall: type: number required: - f1 - precision - recall rouge_l: type: object properties: f1: type: number precision: type: number recall: type: number required: - f1 - precision - recall required: - rouge_1 - rouge_2 - rouge_l required: - type - value title: Rouge N - type: object properties: type: type: string enum: - bert_score value: type: object properties: f1: type: number precision: type: number recall: type: number required: - f1 - precision - recall required: - type - value title: BERT Score - type: object properties: type: type: string enum: - llm_evaluator value: type: - object - 'null' properties: workflow_run_id: type: string trace_id: type: string value: anyOf: - type: number - type: boolean - type: string - type: 'null' explanation: type: - string - 'null' original_value: anyOf: - type: number - type: boolean - type: string - type: 'null' original_explanation: type: - string - 'null' jury: type: object properties: judges_configured: type: integer minimum: 0 judges_succeeded: type: integer minimum: 0 judges_failed: type: integer minimum: 0 replacements_used: type: integer minimum: 0 tie: type: boolean votes: type: array items: type: object properties: model: type: string replacement: type: boolean success: type: boolean value: anyOf: - type: number - type: boolean - type: string explanation: type: string error: type: string required: - model - replacement - success stats: type: object properties: mean: type: number std: type: number required: - mean - std required: - judges_configured - judges_succeeded - judges_failed - replacements_used - tie - votes required: - value required: - type - value title: LLM - type: object properties: type: type: string enum: - http_eval value: type: - object - 'null' properties: type: type: string value: anyOf: - type: number - type: boolean required: - type - value required: - type title: HTTP - type: object properties: type: type: string enum: - structured value: type: object additionalProperties: {} required: - type - value title: Structured '400': description: Bad request content: application/json: schema: type: object properties: message: type: string required: - message '404': description: Workspace ID is not found on the request content: application/json: schema: type: object properties: message: type: string required: - message '500': description: Error running the evaluator content: application/json: schema: type: object properties: message: type: string required: - message components: securitySchemes: ApiKey: type: http scheme: bearer bearerFormat: JWT ````