Skip to main content
Human Review is now configured at the project level for better organization and control. If you previously configured workspace-level human reviews, they will continue to work but are soon deprecated.
Unified Annotations System: Human Reviews are now part of the Annotations system. Use the Annotations API to programmatically apply human reviews to traces, or use Annotation Queues to manage human review workflows in the UI.

​
What is a Human Review

Human Reviews let you create custom fields for structured annotations on model generations. Human Reviews define the schema and validation rules for annotations, which can be applied through the UI or programmatically via the Annotations API.

​
Creating a Human Review

To create a Human Review, head to the Project Settings > Human Review. Add a Human Review pressing the + button.
There are 3 human review types available:
  • Categorical: Create button options with your own labels, such as good/bad or saved/deleted.
  • Range: Set up a custom scoring slider (for example, a scale from 0 to 100).
  • Open field: Allow users to enter free-form feedback, perfect for more detailed comments.

​
Human Review Sets

Human Review Sets are ways to group multiple human review fields together and assign them to specific traces based on Span Type or Name filters. This allows you to create targeted feedback collection workflows that automatically apply the appropriate review fields to different types of AI operations in your system.

​
Creating Human Review Sets

To create a Human Review Set:
  1. Navigate to Project Settings > Human Review
  2. Click the Review Sets tab
  3. Click the + button to create a new set
  1. Configure your review set:
    • Name: Give your set a descriptive name (e.g., “LLM Generation Quality”).
    • Description: Add details about the set’s purpose.
    • Human Reviews: Select which human review fields to include in this set.
    • Filter By: Configure span type or name-based targeting
      • Span types can, for example, be Agent, Deployment, Embedding, Evaluator, Retrieval, or Tool.

​
Using Human Review

Once created, a human review is available within all Feedback forms when viewing Logs for a generation. To learn more: