> ## Documentation Index > Fetch the complete documentation index at: https://docs.orq.ai/llms.txt > Use this file to discover all available pages before exploring further. > Create a new evaluator to assess LLM outputs. Configure scoring criteria, models, and thresholds for automated quality assessment of AI responses. # Create an Evaluator ## OpenAPI ````yaml post /v2/evaluators openapi: 3.1.0 info: title: orq.ai API version: '2.0' description: orq.ai API documentation servers: - url: https://api.orq.ai security: - ApiKey: [] tags: - name: Guardrail Rules - name: Policies - name: Routing Rules - name: Files description: File upload and retrieval operations. - name: FilesService - name: Projects description: Projects organize resources within a workspace - name: ProjectsService - name: Skills description: >- Skills are modular instructions you can use to codify processes and conventions - name: SkillsService - name: Responses - description: >- Run agents on a cadence — cron, interval, or one-off. Minimum firing interval is 1 hour. name: Agent Schedules - name: Reporting description: >- GenAI reporting API over canonical analytics rollups. Accepts a metric name, time range, grain, group-by, and filters; returns a typed time series and optional totals. - name: ReportingService description: |- ReportingService exposes a single QueryReport RPC that maps allowlisted analytics payloads onto safe rollup queries. Callers never send SQL; the backend picks the rollup family and grain from the metric catalogue, the requested range, and the requested grouping. externalDocs: url: https://docs.orq.ai description: orq.ai Documentation paths: /v2/evaluators: post: tags: - Evals summary: Create an Evaluator operationId: CreateEval requestBody: content: application/json: schema: oneOf: - oneOf: - type: object properties: guardrail_config: oneOf: - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - enabled - type - value title: Boolean - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - enabled - type - values title: Categorical - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - enabled - type - value - operator title: Number - type: 'null' output_type: type: string enum: - boolean - categorical - number - string description: The type of output expected from the evaluator type: type: string enum: - llm_eval repetitions: type: integer minimum: 1 maximum: 3 prompt: type: string categories: type: array items: type: string categorical_labels: type: array items: type: object properties: value: type: string minLength: 1 description: type: string required: - value path: type: string description: >- Entity storage path. With workspace-level API keys, use the format `project/folder/subfolder/...`. The first element identifies the project, followed by nested folders (auto-created as needed). Example: `Default/agents`. With project-level API keys, the project is predetermined by the API key, so the path is relative to that project. Example: `agents`. For backward compatibility, a leading project name is ignored when it matches the scoped project. example: Default description: type: string default: '' key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ mode: type: string enum: - single model: type: string required: - type - prompt - path - key - mode - model - type: object properties: guardrail_config: oneOf: - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - enabled - type - value title: Boolean - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - enabled - type - values title: Categorical - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - enabled - type - value - operator title: Number - type: 'null' output_type: type: string enum: - boolean - categorical - number - string description: The type of output expected from the evaluator type: type: string enum: - llm_eval repetitions: type: integer minimum: 1 maximum: 3 prompt: type: string categories: type: array items: type: string categorical_labels: type: array items: type: object properties: value: type: string minLength: 1 description: type: string required: - value path: type: string description: >- Entity storage path. With workspace-level API keys, use the format `project/folder/subfolder/...`. The first element identifies the project, followed by nested folders (auto-created as needed). Example: `Default/agents`. With project-level API keys, the project is predetermined by the API key, so the path is relative to that project. Example: `agents`. For backward compatibility, a leading project name is ignored when it matches the scoped project. example: Default description: type: string default: '' key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ mode: type: string enum: - jury jury: type: object properties: judges: type: array items: type: object properties: model: type: string retry: type: object properties: count: type: integer minimum: 1 maximum: 5 default: 2 on_codes: type: array items: type: integer minimum: 100 maximum: 599 minItems: 1 default: - 429 - 500 - 502 - 503 - 504 fallbacks: type: array items: type: object properties: model: type: string required: - model default: [] required: - model minItems: 2 replacement_judges: type: array items: type: object properties: model: type: string retry: type: object properties: count: type: integer minimum: 1 maximum: 5 default: 2 on_codes: type: array items: type: integer minimum: 100 maximum: 599 minItems: 1 default: - 429 - 500 - 502 - 503 - 504 fallbacks: type: array items: type: object properties: model: type: string required: - model default: [] required: - model default: [] min_successful_judges: type: integer minimum: 2 default: 2 tie_value: type: string enum: - Tie default: Tie required: - judges required: - type - prompt - path - key - mode - jury title: LLM - type: object properties: guardrail_config: oneOf: - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - enabled - type - value title: Boolean - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - enabled - type - values title: Categorical - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - enabled - type - value - operator title: Number - type: 'null' output_type: type: string enum: - boolean type: enum: - json_schema type: string schema: type: string path: type: string description: >- Entity storage path. With workspace-level API keys, use the format `project/folder/subfolder/...`. The first element identifies the project, followed by nested folders (auto-created as needed). Example: `Default/agents`. With project-level API keys, the project is predetermined by the API key, so the path is relative to that project. Example: `agents`. For backward compatibility, a leading project name is ignored when it matches the scoped project. example: Default description: type: string default: '' key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ required: - type - schema - path - key title: JSON - type: object properties: guardrail_config: oneOf: - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - enabled - type - value title: Boolean - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - enabled - type - values title: Categorical - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - enabled - type - value - operator title: Number - type: 'null' output_type: type: string enum: - boolean - categorical - number - string description: The type of output expected from the evaluator type: type: string enum: - http_eval url: type: string method: type: string enum: - GET - POST headers: type: object additionalProperties: type: string payload: type: object additionalProperties: {} path: type: string description: >- Entity storage path. With workspace-level API keys, use the format `project/folder/subfolder/...`. The first element identifies the project, followed by nested folders (auto-created as needed). Example: `Default/agents`. With project-level API keys, the project is predetermined by the API key, so the path is relative to that project. Example: `agents`. For backward compatibility, a leading project name is ignored when it matches the scoped project. example: Default description: type: string default: '' key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ required: - type - url - method - headers - payload - path - key title: HTTP - type: object properties: guardrail_config: oneOf: - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - enabled - type - value title: Boolean - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - enabled - type - values title: Categorical - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - enabled - type - value - operator title: Number - type: 'null' output_type: type: string enum: - boolean - number code: type: string type: type: string enum: - python_eval path: type: string description: >- Entity storage path. With workspace-level API keys, use the format `project/folder/subfolder/...`. The first element identifies the project, followed by nested folders (auto-created as needed). Example: `Default/agents`. With project-level API keys, the project is predetermined by the API key, so the path is relative to that project. Example: `agents`. For backward compatibility, a leading project name is ignored when it matches the scoped project. example: Default description: type: string default: '' key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ required: - code - type - path - key title: Python responses: '200': description: Successfully created an evaluator content: application/json: schema: oneOf: - oneOf: - type: object properties: _id: type: string description: type: string created: type: string default: '2026-05-14T21:19:48.442Z' updated: type: string default: '2026-05-14T21:19:48.442Z' guardrail_config: oneOf: - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - enabled - type - value title: Boolean - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - enabled - type - values title: Categorical - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - enabled - type - value - operator title: Number - type: 'null' type: type: string enum: - llm_eval repetitions: type: integer minimum: 1 maximum: 3 prompt: type: string categories: type: array items: type: string categorical_labels: type: array items: type: object properties: value: type: string minLength: 1 description: type: string required: - value key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ mode: type: string enum: - single model: type: string required: - _id - description - type - prompt - key - mode - model - type: object properties: _id: type: string description: type: string created: type: string default: '2026-05-14T21:19:48.442Z' updated: type: string default: '2026-05-14T21:19:48.442Z' guardrail_config: oneOf: - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - enabled - type - value title: Boolean - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - enabled - type - values title: Categorical - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - enabled - type - value - operator title: Number - type: 'null' type: type: string enum: - llm_eval repetitions: type: integer minimum: 1 maximum: 3 prompt: type: string categories: type: array items: type: string categorical_labels: type: array items: type: object properties: value: type: string minLength: 1 description: type: string required: - value key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ mode: type: string enum: - jury jury: type: object properties: judges: type: array items: type: object properties: model: type: string retry: type: object properties: count: type: integer minimum: 1 maximum: 5 default: 2 on_codes: type: array items: type: integer minimum: 100 maximum: 599 minItems: 1 default: - 429 - 500 - 502 - 503 - 504 fallbacks: type: array items: type: object properties: model: type: string required: - model default: [] required: - model minItems: 2 replacement_judges: type: array items: type: object properties: model: type: string retry: type: object properties: count: type: integer minimum: 1 maximum: 5 default: 2 on_codes: type: array items: type: integer minimum: 100 maximum: 599 minItems: 1 default: - 429 - 500 - 502 - 503 - 504 fallbacks: type: array items: type: object properties: model: type: string required: - model default: [] required: - model default: [] min_successful_judges: type: integer minimum: 2 default: 2 tie_value: type: string enum: - Tie default: Tie required: - judges required: - _id - description - type - prompt - key - mode - jury title: LLM - type: object properties: _id: type: string description: type: string created: type: string default: '2026-05-14T21:19:48.442Z' updated: type: string default: '2026-05-14T21:19:48.442Z' guardrail_config: oneOf: - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - enabled - type - value title: Boolean - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - enabled - type - values title: Categorical - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - enabled - type - value - operator title: Number - type: 'null' type: enum: - json_schema type: string schema: type: string key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ required: - _id - description - type - schema - key title: JSON - type: object properties: _id: type: string description: type: string created: type: string default: '2026-05-14T21:19:48.442Z' updated: type: string default: '2026-05-14T21:19:48.442Z' guardrail_config: oneOf: - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - enabled - type - value title: Boolean - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - enabled - type - values title: Categorical - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - enabled - type - value - operator title: Number - type: 'null' type: type: string enum: - http_eval url: type: string method: type: string enum: - GET - POST headers: type: object additionalProperties: type: string payload: type: object additionalProperties: {} key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ required: - _id - description - type - url - method - headers - payload - key title: HTTP - type: object properties: _id: type: string description: type: string created: type: string default: '2026-05-14T21:19:48.442Z' updated: type: string default: '2026-05-14T21:19:48.442Z' guardrail_config: oneOf: - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - enabled - type - value title: Boolean - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - enabled - type - values title: Categorical - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - enabled - type - value - operator title: Number - type: 'null' code: type: string type: type: string enum: - python_eval key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ required: - _id - description - code - type - key title: Python - type: object properties: _id: type: string description: type: string created: type: string default: '2026-05-14T21:19:48.442Z' updated: type: string default: '2026-05-14T21:19:48.442Z' guardrail_config: oneOf: - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - enabled - type - value title: Boolean - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - enabled - type - values title: Categorical - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - enabled - type - value - operator title: Number - type: 'null' type: type: string enum: - function_eval function_params: oneOf: - type: object properties: type: type: string enum: - contains value: type: string required: - type - value - type: object properties: type: type: string enum: - contains_none keywords: type: array items: type: string required: - type - keywords - type: object properties: type: type: string enum: - contains_all keywords: type: array items: type: string required: - type - keywords - type: object properties: type: type: string enum: - contains_any keywords: type: array items: type: string required: - type - keywords - type: object properties: type: type: string enum: - contains_email required: - type - type: object properties: type: type: string enum: - contains_url required: - type - type: object properties: type: type: string enum: - contains_valid_link required: - type - type: object properties: type: type: string enum: - start_with value: type: string required: - type - value - type: object properties: type: type: string enum: - ends_with value: type: string required: - type - value - type: object properties: type: type: string enum: - exact_match required: - type - type: object properties: type: type: string enum: - length_less_than value: type: number required: - type - value - type: object properties: type: type: string enum: - length_between min: type: number max: type: number required: - type - min - max - type: object properties: type: type: string enum: - length_greater_than value: type: number required: - type - value - type: object properties: type: type: string enum: - one_line required: - type - type: object properties: type: type: string enum: - regex pattern: type: string required: - type - pattern - type: object properties: type: type: string enum: - is_valid_json required: - type - type: object properties: type: type: string enum: - moderations_openai required: - type - type: object properties: type: type: string enum: - moderations_google required: - type - type: object properties: type: type: string enum: - bert_score required: - type - type: object properties: type: type: string enum: - bleu_score required: - type - type: object properties: type: type: string enum: - rouge_n required: - type - type: object properties: type: type: string enum: - meteor_score required: - type - type: object properties: type: type: string enum: - cosine_similarity required: - type - type: object properties: type: type: string enum: - levenshtein_distance required: - type - type: object properties: type: type: string enum: - keywords_match keywords: type: array items: type: string required: - type - keywords - type: object properties: type: type: string enum: - most_repeated_words required: - type - type: object properties: type: type: string enum: - flesch_reading_ease required: - type - type: object properties: type: type: string enum: - gse_english_level required: - type - type: object properties: type: type: string enum: - words_count required: - type - type: object properties: type: type: string enum: - sentences_count required: - type - type: object properties: type: type: string enum: - lexical_repetition required: - type - type: object properties: type: type: string enum: - grammar_diversity required: - type key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ required: - _id - description - type - function_params - key title: Function - type: object properties: _id: type: string description: type: string created: type: string default: '2026-05-14T21:19:48.442Z' updated: type: string default: '2026-05-14T21:19:48.442Z' guardrail_config: oneOf: - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - enabled - type - value title: Boolean - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - enabled - type - values title: Categorical - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - enabled - type - value - operator title: Number - type: 'null' type: type: string enum: - ragas ragas_metric: type: string enum: - context_precision - context_recall - context_entities_recall - harmfulness - maliciousness - coherence - correctness - conciseness - response_relevancy - faithfulness - summarization - noise_sensitivity key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ model: type: string required: - _id - description - type - ragas_metric - key - model title: Ragas - type: object properties: _id: type: string description: type: string created: type: string default: '2026-05-14T21:19:48.442Z' updated: type: string default: '2026-05-14T21:19:48.442Z' guardrail_config: oneOf: - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - boolean value: type: boolean required: - enabled - type - value title: Boolean - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - categorical values: type: array items: type: string minLength: 1 required: - enabled - type - values title: Categorical - type: object properties: enabled: type: boolean alert_on_failure: type: boolean default: false type: type: string enum: - number value: type: number operator: type: string enum: - eq - ne - gt - gte - lt - lte required: - enabled - type - value - operator title: Number - type: 'null' code: type: string type: type: string enum: - typescript_eval key: type: string minLength: 1 pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$ required: - _id - description - code - type - key title: Typescript '403': description: Domain is not allowed '404': description: Workspace ID is not found on the request content: application/json: schema: type: object properties: message: type: string required: - message components: securitySchemes: ApiKey: type: http scheme: bearer bearerFormat: JWT ````