Evaluators API

We’ve expanded evaluator support with two new APIs, enabling much more flexible, code-driven workflows:

Evaluators Hub API: Run any evaluator on demand via API, without needing to attach it to an experiment or deployment. Ideal for integrating evaluator runs into your own pipelines or automated tools.
Evaluators API: Create, retrieve, update, and delete evaluators directly via API. This allows you to fully manage your evaluators programmatically, instead of only through the UI.

You can use both APIs together to build and manage evaluators entirely through code, combining on-demand runs with dynamic evaluator management. This enables teams to automate their evaluation workflows, integrate with CI/CD, and adapt evaluators to changing requirements—all without leaving their own codebase.

Key benefits:

Decoupled Evaluation: Run and manage evaluators without relying on experiments or deployments.
Full API Control: Create, update, and remove evaluators as your evaluation needs evolve.
Pipeline Integration: Integrate both evaluator management and execution into your own data pipelines or CI/CD workflows.
Faster Iteration: Quickly test, refine, and iterate on evaluators using code.
Automation: Scale evaluations and management as needed for batch operations or continuous quality monitoring.

See an example on how the new Evaluator API works with RAGAS faithfulness below

curl --request POST \
     --url https://api.orq.ai/v2/evaluators/ragas_faithfulness \
     --header 'accept: application/json' \
     --header 'authorization: Bearer $ORQ_API_KEY' \
     --header 'content-type: application/json' \
     --data '
{
  "model": "openai/gpt-4.1",
  "query": "How large can my carry on bag be?",
  "output": "Your carry-on bag must not exceed 22 x 14 x 9 inches (56 x 36 x 23 cm), including handles and wheels. There is no weight limit for carry-on bags, but you must be able to lift it into the overhead bin without assistance.",
  "retrievals": [
    "{ \"question\": \"What are the size and weight limits for carry-on bags?\", \"answer\": \"Your carry-on bag must not exceed 22 x 14 x 9 inches (56 x 36 x 23 cm), including handles and wheels. There is no weight limit for carry-on bags, but you must be able to lift it into the overhead bin without assistance.\" }",
    "{ \"question\": \"Are there bag or seasonal bag limitations?\", \"answer\": \"Learn about checked bag limitations on our website.\" }",
    "{ \"question\": \"What are your liability limitations for lost bags?\", \"answer\": \"Learn about our liability limitations and see items excluded from liability on our website.\" }",
    "{ \"question\": \"What should I do about delayed or damaged bags?\", \"answer\": \"Find out what to do about delayed or damaged bags on our website.\" }"
  ]
}
'