Setting up an HTTP Evaluator

To start building an HTTP Evaluator head to the Orq.ai panel, then to the Evaluators sections.

Select Create Evaluator, the following modal opens:

You can then define the details of your Evaluator:

Expected Response Payload

For an HTTP Evaluator to be valid, Orq.ai is expecting a certain response payload returning the evaluation result.

Boolean Response

You can decide to return a Boolean response to the evaluation, the following is the expected payload:

{
    "type": "boolean",
    "value": true
}

Number Response

You can decide to return a Number response to the evaluation, the following is the expected payload:

{
    "type": "number",
    "value": 1
}

If you fail to return one of the two payloads shown above, the Evaluator will be ignored during processing.

Example for HTTP Evaluators

HTTP Evaluators can be useful to implement business or industry-specific checks from within your applications. You can build an Evaluator using an API on your systems that will perform a compliance check for instance.

This HTTP Evaluator has agency over calls routed through Orq.ai while keeping business intelligence and logic within your environments. This ensures that generated content adheres to their specific regulatory guidelines.

For example, in case the content is not adhering to regulatory guidelines, the HTTP call could return the following, failing the evaluator along the way.

{
    "type": "boolean",
    "value": true
}

Guardrail Configuration

Within a Deployments, you can use your HTTP Evaluator as a Guardrail, effectively preventing deployments to respond to a user depending on the Input or Output.

Here you can define the guardrail condition:

• if the HTTP Evaluator returns a value higher than the defined value, the call is accepted.
• If the HTTP Evaluator returns a value lower or equal to the defined value, the call is then denied.

Once created the Evaluator will be available to use in Experiments and Deployments, to learn more see Using an HTTP Evaluator.