Evaluator

Evaluators are automated tools designed to assess the performance and outputs of models within an Experiment or Deployment.

Evaluators can verify outputs against reference data, ensure compliance with specific criteria, and perform various automated validations.

By utilizing Evaluators, teams can automate the validation process, maintain high-quality outputs, and ensure that their AI systems operate within desired parameters.

To create started:

Create an Evaluator

• Create an Evaluator via the Studio
• Creating an Evaluator via the API

Use an Evaluator

• Using Evaluator in Experiment
• Evaluators & Guardrails in Deployments

Evaluator Library

• Import existing Evaluators from Evaluator Library in the Hub
• Use the Evaluator Library via the API
• Learn more about our different Evaluator types:
  • Function Evaluator readily available in the Hub
  • HTTP Evaluator
  • JSON Evaluator
  • LLM Evaluator readily available in the Hub
  • Python Evaluator
  • Ragas Evaluator readily available in the Hub