What are Annotations in the AI Studio?
Annotations allow you to capture structured human feedback on AI responses directly in the AI Studio. Domain experts and reviewers can annotate traces with quality assessments, corrections, and custom feedback to improve your AI applications over time.Adding Annotations
The annotation capabilities differ between Logs and Traces. Logs support both human feedback and corrections, while Traces only support human feedback annotations.- Traces
- Logs
Navigate to the Traces view and select a single trace. The Annotations panel will be displayed, allowing you to apply human feedback to the AI response.
Using Human Reviews
Annotations are based on Human Review definitions that you create. Human Reviews define the structure and validation rules for your annotations:- Key: Unique identifier for the annotation (e.g., “rating”, “defects”)
- Type: The data type (string, number, or array of strings)
- Options: Available choices for select-type annotations
Create Human Reviews in Project Settings > Human Review before adding annotations. Each annotation key must match an existing Human Review.
Common Annotation Types
Rating
Rate the overall quality of AI responses:| Rating | Description |
|---|---|
| good | The response was helpful and accurate. |
| bad | The response was unhelpful or inaccurate. |
Defects
Identify specific issues with AI responses:| Defect | Description |
|---|---|
| grammatical | Responses that contain grammatical errors |
| spelling | Responses that contain spelling errors |
| hallucination | Responses that contain hallucinations or factual inaccuracies |
| repetition | Responses that contain unnecessary repetition |
| inappropriate | Responses that are deemed inappropriate or offensive |
| off_topic | Responses that do not address the user’s query |
| incompleteness | Responses that are incomplete or partially address the query |
| ambiguity | Responses that are vague or unclear |
You can select multiple defects for one response by using an array-type Human Review.
Custom Annotations
You can create custom annotation types for your specific use cases by defining custom Human Reviews. This allows you to track domain-specific quality metrics, compliance requirements, or any other human feedback relevant to your AI application.Programmatic Annotations
Annotations can also be submitted programmatically via the API. Learn more in Annotations via the API.