Setting up a JSON Evaluators

To start building an JSON Evaluator head to the Orq.ai panel, then to the Evaluators sections.

Select Create Evaluator, the following modal opens:

You can then enter a JSON Schema that the input or output will be validated against during Deployment generation.

A JSON Schema lets you define which fields you want to find in the evaluated payload

Here is an example defining two mandatory fields: title and length

{
  "type": "object",
  "properties": {
    "title": {
      "type": "string",
      "description": "The post title"
    },
    "length": {
      "type": "integer"
      "description": "The post length"
    },
  }
  "required": [ "title", "length" ]
}

Guardrail Configuration

Within a Deployments, you can use your JSON Evaluator as a Guardrail, effectively permitting a JSON Validation on input and output for a deployment generation.

Enabling the Guardrail toggle will block payloads that don't validate the given JSON Schema.

Once created the Evaluator will be available to use in Deployments, to learn more see Using a JSON Evaluator