HTTP and JSON Evaluators and Guardrails

You can now add HTTP and JSON Evaluators and Guardrails under the Evaluator tab and add them to your Deployment or Experiment.

HTTP Evaluators and Guardrails

• Purpose: Enable evaluation using external APIs to perform checks like compliance, custom logic, or business-specific validations.
• How It Works: You define the URL, headers, and payload for HTTP Requests, and the evaluator expects either a boolean or numerical response in the format:

{ "type": "boolean", "value": true }

{ "type": "number", "value": 1 }

• Use Case Example: An HTTP Evaluator can validate content against regulatory guidelines by routing a compliance check to your API, returning a boolean to accept or block the output. If you want to integrate an HTTP Eval into your set up, you can find more information in HTTP Evaluators.
• Guardrails: Configurable to accept or deny calls based on the returned value, providing dynamic control in experiments and deployments.

JSON Evaluators and Guardrails

• Purpose: Validate input or output payloads during Experiments or Deployments using a predefined JSON Schema.
• How It Works: Enter a JSON Schema specifying required fields and their expected types. For example:

{
  "name": "ReceiptDataExtraction",
  "strict": true,
  "schema": {
    "type": "object",
    "properties": {
      "Date": {
        "type": "string",
        "description": "Transaction date (YYYY-MM-DD)."
      },
      "Vendor": {
        "type": "string",
        "description": "Name of the vendor."
      },
      "Amount": {
        "type": "number",
        "description": "Total transaction amount."
      }
    },
    "required": ["Date", "Vendor", "Amount"],
    "additionalProperties": false
  }
}

• Use Case Example: A JSON Evaluator can ensure that generated content includes mandatory fields like "title" and "length," blocking invalid payloads. For a more extensive explanation on how to set this up in your Deployment, you can go here.
• Guardrails: Toggle guardrails to block outputs that don’t adhere to the schema, ensuring compliance during Deployments or Experiments.