Skip to main content

​
Prerequisite

To get started, an API key is needed to use within SDKs or HTTP API.
To get an API key ready, see Authentication.

​
SDK

Node.js

Python

​
Example

We’ll be calling the Tone of Voice endpoint: Here is an example call:
The query defines the way the evaluator runs on the given output. 
curl --request POST \
     --url https://api.orq.ai/v2/evaluators/tone_of_voice \
     --header 'accept: application/json' \
     --header 'authorization: Bearer <ORQ_API_KEY>' \
     --header 'content-type: application/json' \
     --data '
{
  "query": "Validate the tone of voice if it is professional.",
  "output": "Hello, how are you ??",
  "model": "openai/gpt-4o"
}
'
Here is the result returned by the API
The value here holds result of the evaluator call following the query 
{
  "value": {
    "value": false,
    "explanation": "The output does not align with a professional tone. The use of 'Hello, how are you ??' is informal and lacks the formality expected in professional communication. The double question marks and casual greeting are more suited to a casual or friendly context rather than a professional one. A professional tone would require a more formal greeting and a clear purpose for the communication."
  }
}

​
Calling a custom evaluator

It is also possible to call a custom-made Evaluator made on orq using the API. You can fetch the Evaluator ID to send to this call by searching for Evaluators using the Get all Evaluators API. Then you can run the following API call: 
curl 'https://my.orq.ai/v2/evaluators/<evaluator_id>/invoke' \
-H 'Authorization: Bearer <ORQ_API_KEY>' \
-H 'Content-Type: application/json' \
-H 'Accept: application/json' \
--data-raw '{
    "query": "Your input text",
    "output": "Your output text",
    "reference": "Optional reference text",
    "messages": [
        {
            "role": "user",
            "content": "Your message"
        }
    ],
    "retrievals": ["Your retrieval content"]
}' \
--compressed
Finally, through the Orq studio, find the View Code button on your Evaluator page. the following modal opens:

Evaluator code is available directly to use.

This code is used to call the current Evaluator through the API. Ensure the payload is containing all necessary data for the Evaluator to execute correctly.

​
Using EvaluatorQ

EvaluatorQ is a dedicated SDK for using Evaluators within your application. It features the following capabilities:
  • Parallel Execution: Run multiple evaluation jobs concurrently with progress tracking
  • Flexible Data Sources: Support for inline data, promises, and Orq platform datasets
  • Type-safe: Fully written in TypeScript
Installation: 
npm install @orq-ai/evaluatorq
Usage example: 
import { evaluatorq, job } from "@orq-ai/evaluatorq";

const textAnalyzer = job("text-analyzer", async (data) => {
  const text = data.inputs.text;
  const analysis = {
    length: text.length,
    wordCount: text.split(" ").length,
    uppercase: text.toUpperCase(),
  };

  return analysis;
});

await evaluatorq("text-analysis", {
  data: [
    { inputs: { text: "Hello world" } },
    { inputs: { text: "Testing evaluation" } },
  ],
  jobs: [textAnalyzer],
  evaluators: [
    {
      name: "length-check",
      scorer: async ({ output }) => {
        const passesCheck = output.length > 10;
        return {
          value: passesCheck ? 1 : 0,
          explanation: passesCheck
            ? "Output length is sufficient"
            : `Output too short (${output.length} chars, need >10)`,
        };
      },
    },
  ],
});
Learn more, see the EvaluatorQ repository on Github