Creating a Python Evaluator

To start building a Python Evaluator , head to a Projects, use the + button and select Evaluator.

The following modal opens:

You'll be then taken to the code editor to configure your Python evaluation.

To perform an evaluation, you have access to the log of the Evaluated Model, which contains the following three fields:

• log["input"] <str> The last message sent to generate the output.
• log["output"] <str> The generated response from the model.
• log["reference"] <str> The reference used to compare the output.
• log["messages"] list<str> All previous messages sent to the model.
• log["retrievals"] list<str> All Knowledge Base retrievals.

The evaluator can be configured with two different response types:

• Number to return a score
• Boolean to return a true/false value

The following example compares the output size with the given reference.

def evaluate(log):
    output_size = len(log["output"])
    reference_size = len(log["reference"])
    return abs(output_size - reference_size)

📘

You can define multiple methods within the code editor, the last method will be the entry-point for the Evaluator when run.

Environment and Libraries

The Python Evaluator runs in the following environment: python 3.11

The environment comes preloaded with the following libraries:

numpy==1.26.4

Guardrail Configuration

Within a Deployment, you can use your Python Evaluator as a Guardrail, blocking potential calls to

Enabling the Guardrail toggle will block payloads that don't validate the given JSON Schema.

Once created the Evaluator will be available to use in Deployments, to learn more see Evaluators & Guardrails in Deployments.