Evaluators and Guardrails in Deployments

In Deployments Settings it is possible to set Evaluators on the Inputs and Outputs of a Generation.

Workflow

Guardrails will be executed synchronously if they exists while Evaluators will be non-blocking to ensure quickest response-time to the user.

Input & Output Evaluators

You can add Evaluators available in your Library as Input Evaluator for a Deployment.

When adding an Evaluator here, you can intercept and evaluate asynchronously the Input sent to the configured model or the Ouput generated and sent back to the user.

You can configure a Sample Rate (percentage) and define the frequency at which the evaluator will be used.

The Sample Rate goes from 0 (0%) to 100 (100%) rate.

Guardrails

If an Evaluators has a Guardrail capability, it can be used as an Input Guardrail or Output Guardrail in a Deployment.

A Guardrail will effectively Deny a generation if its evaluation fails.In this case, an error will be sent back to the user.

Guardrail Behaviour

You can decide here two behaviour in case your Guardrail does not pass on a generation:

Retry the current Generation
use the Fallback behaviour, if the configured model has a fallback configured, the call will be tried there.