Skip to main content
POST
/
v2
/
evaluators
Create an Evaluator
curl --request POST \
  --url https://api.orq.ai/v2/evaluators \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "type": "llm_eval",
  "prompt": "<string>",
  "path": "Default",
  "key": "<string>",
  "mode": "single",
  "model": "<string>",
  "guardrail_config": {
    "type": "boolean",
    "value": true,
    "enabled": true,
    "alert_on_failure": false
  },
  "repetitions": 2,
  "categories": [
    "<string>"
  ],
  "categorical_labels": [
    {
      "value": "<string>",
      "description": "<string>"
    }
  ],
  "dataset_id": "<string>",
  "description": ""
}
'
{
  "_id": "<string>",
  "description": "<string>",
  "type": "llm_eval",
  "prompt": "<string>",
  "key": "<string>",
  "mode": "single",
  "model": "<string>",
  "created": "2026-06-10T10:08:15.342Z",
  "updated": "2026-06-10T10:08:15.342Z",
  "updated_by_id": "<string>",
  "guardrail_config": {
    "type": "boolean",
    "value": true,
    "enabled": true,
    "alert_on_failure": false
  },
  "repetitions": 2,
  "categories": [
    "<string>"
  ],
  "categorical_labels": [
    {
      "value": "<string>",
      "description": "<string>"
    }
  ],
  "dataset_id": "<string>"
}

Authorizations

​
Authorization
string
header
required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Body

application/json
​
type
enum<string>
required
Available options:
llm_eval
​
prompt
string
required
​
path
string
required

Entity storage path.

With workspace-level API keys, use the format project/folder/subfolder/.... The first element identifies the project, followed by nested folders (auto-created as needed). Example: Default/agents.

With project-level API keys, the project is predetermined by the API key, so the path is relative to that project. Example: agents. For backward compatibility, a leading project name is ignored when it matches the scoped project.

Example:

"Default"

​
key
string
required
Minimum string length: 1
Pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$
​
mode
enum<string>
required
Available options:
single
​
model
string
required
​
guardrail_config
Boolean · object
​
output_type
enum<string>

The type of output expected from the evaluator

Available options:
boolean,
categorical,
number,
string
​
repetitions
integer | null
Required range: 1 <= x <= 3
​
categories
string[] | null
​
categorical_labels
object[] | null
​
dataset_id
string
​
description
string
default:""

Response

Successfully created an evaluator

​
_id
string
required
​
description
string
required
​
type
enum<string>
required
Available options:
llm_eval
​
prompt
string
required
​
key
string
required
Minimum string length: 1
Pattern: ^[a-zA-Z0-9]([a-zA-Z0-9_-]*[a-zA-Z0-9])?$
​
mode
enum<string>
required
Available options:
single
​
model
string
required
​
created
string
default:2026-06-10T10:08:15.342Z
​
updated
string
default:2026-06-10T10:08:15.342Z
​
updated_by_id
string | null
​
guardrail_config
Boolean · object
​
repetitions
integer | null
Required range: 1 <= x <= 3
​
categories
string[] | null
​
categorical_labels
object[] | null
​
dataset_id
string