Evaluator Library

The Evaluator Library shows all the available Evaluators usable in Experiments and Deployments.

You can create Evaluators either by using a Template from the Gallery or by Creating a custom one to fit your need.

Evaluator Templates Gallery

The Evaluator Gallery is accessible by selecting the Gallery tab. Here you can see all pre-built Evaluators Templates to choose from. You can read description details for each evaluator to see how they can be used in your environments.

Adding an Evaluator Template from the Gallery the Library

To add an Evaluator Template from the Gallery to the Library, select the + button.

You can then customize the Template:

Model Configuration

For certain Templates, you need to configure a compatible model. If you don't have any compatible model available, add one using your Model Garden.

Guardrail Configuration

Within Deployments, an Evaluator can be used to prevent requests to be processed or returned to users. To do so you need to Toggle the Guardrail Configuration toggle and set the Value for which you would want generations to be passed.

📘

Guardrails are defined in Deployments, to learn more see Evaluators and Guardrails in Deployments.

Once the model configuration is completed select Create. The model will then be available in your Library and up for integration in Experiments and Deployments.

Creating a Custom Evaluator

You can create multiple types of custom Evaluators. Select the Create button and choose from the different types available:

• API Call, to generate an API integrated evaluator, to learn more see Setting up an HTTP Evaluator.
• JSON Evaluator, to generate JSON validations through evaluator, to learn more see Setting up a JSON Evaluators
• LLM as a Judge, to use an LLM model to perform evaluation, to learn more see Custom LLM Evaluators.