Skip to main content
You can also create an Evaluator using the API, see Creating an Evaluator via the API.
To start building an JSON Evaluator, head to a Projects, use the + button and select Evaluator. The following modal opens:

Select the Evaluator type

You can then enter a JSON Schema that the input or output will be validated against during Deployment generation.
A JSON Schema lets you define which fields you want to find in the evaluated payload Here is an example defining two mandatory fields: title and length 
{
  "type": "object",
  "properties": {
    "title": {
      "type": "string",
      "description": "The post title"
    },
    "length": {
      "type": "integer"
      "description": "The post length"
    },
  }
  "required": [ "title", "length" ]
}

​
Testing an Evaluator

Within the Studio, a Playground is available to test an evaluator against any output. This helps validates quickly that an evaluator is behaving correctly To do so, first configure the request:

Here you can configure the JSON payload that will be sent to an evaluator.

Use the Run button to execute your evaluator with the request payload. The result will be displayed in the Response field.

A JSON test response.

​
Guardrail Configuration

Within a Deployment, you can use your JSON Evaluator as a Guardrail, effectively permitting a JSON Validation on input and output for a deployment generation. Enabling the Guardrail toggle will block payloads that don’t validate the given JSON Schema. Once created the Evaluator will be available to use in Deployments, to learn more see Evaluators & Guardrails in Deployments.
JSON Evaluator LLM Evaluator