Creating an HTTP Evaluator

📘

You can also create an Evaluator using the API, see Creating an Evaluator via the API.

To start building an HTTP Evaluator head to a Project and use + and choose Evaluator

The following modal opens:

You can then define the details of your Evaluator:

Payload Detail

Here are the payload variables accessible to perform evaluation

{
  "query": "", // string containing the last message sent to the model
  "response": "", // string containing the assistant generated response
  "expected_output": "", // string containing the dataset reference for the evaluation
  "retrieved_context": [] // array of strings containing the  knowledge base retrievals
}

Expected Response Payload

For an HTTP Evaluator to be valid, orq.ai is expecting a certain response payload returning the evaluation result.

Boolean Response

You can decide to return a Boolean response to the evaluation, the following is the expected payload:

{
    "type": "boolean",
    "value": true
}

Number Response

You can decide to return a Number response to the evaluation, the following is the expected payload:

{
    "type": "number",
    "value": 1
}

If you fail to return one of the two payloads shown above, the Evaluator will be ignored during processing.

String Response

You can decide to return a String response to the evaluation, the following is the expected payload:

{
  "type": "string",
  "value": "This response passed all compliance checks"
}

Testing an Evaluator

Within the Studio, a Playground is available to test an evaluator against any output. This helps validates quickly that an evaluator is behaving correctly

To do so, first configure the request:

Use the Run button to execute your evaluator with the request payload. The result will be displayed in the Response field.

Example for HTTP Evaluators

HTTP Evaluators can be useful to implement business or industry-specific checks from within your applications. You can build an Evaluator using an API on your systems that will perform a compliance check for instance.

This HTTP Evaluator has agency over calls routed through orq.ai while keeping business intelligence and logic within your environments. This ensures that generated content adheres to their specific regulatory guidelines.

For example, in case the content is not adhering to regulatory guidelines, the HTTP call could return the following, failing the evaluator along the way.

{
    "type": "boolean",
    "value": true
}

Guardrail Configuration

Within a Deployment, you can use your HTTP Evaluator as a Guardrail, effectively preventing deployments to respond to a user depending on the Input or Output.

Here you can define the guardrail condition:

• if the HTTP Evaluator returns a value higher than the defined value, the call is accepted.
• If the HTTP Evaluator returns a value lower or equal to the defined value, the call is then denied.

Once created the Evaluator will be available to use in Experiments and Deployments, to learn more see Using Evaluator in Experiment.