What is Human in the Loop
Human in the Loop is a way for you to collect feedback from your end users and have your domain expert annotate feedback and corrections on each log for future improvement.Logging Feedback
Logging feedback is especially useful for domain experts to validate an output, flag defects, and mark interactions with responses generated by models. Within any module, by choosing the Logs tab and then selecting a single Log, the Feedback panel will be displayed on the right.
Within the Log panel, the following is available on the right, letting you qualify the selected generation.
Feedback Types
There are three different types of feedback available to you in the Feedback Panel, within your Logs.
The Feedback Panel is always available within Logs.
Rating
Rating lets you generally qualify the quality of the response sent, here are the available values:| Rating | Description |
|---|---|
| good | The response was helpful and accurate. |
| bad | The response was unhelpful or inaccurate. |
Defects
Defects let you further define what was wrong within the selected generation, here are the available values:| Defect | Description |
|---|---|
| grammatical | Flag for responses that contain grammatical errors. |
| spelling | Flag for responses that contain spelling errors. |
| hallucination | Flag for responses that contain hallucinations or factual inaccuracies. |
| repetition | Flag for responses that contain unnecessary repetition. |
| inappropriate | Flag for responses that are deemed inappropriate or offensive. |
| off_topic | Flag for responses that do not address the user’s query. |
| incompleteness | Flag for responses that are incomplete or partially address the query. |
| ambiguity | Flag for responses that are vague or unclear. |
You can select multiple defects for one response
Interactions
Interactions let you qualify how the user interacted with the response, here are the available values:| Interaction | Description |
|---|---|
| saved | Indicates if the user saved the response |
| selected | Indicates if the user selected this response from multiple options |
| deleted | Indicates if the user deleted or discarded the response |
| shared | Indicates if the user shared the response with others |
| copied | Indicates if the user copied the response for use elsewhere |
| reported | Indicates if the user reported this response for review |
You can select multiple interactions for one response
Custom Feedback
You can create custom feedback types, to learn more, see Human Review.
Feedbacks can be submitted using the API, to learn more see Adding Feedback Programmatically.
Making Corrections
To add a correction, first find the desired response by choosing the Logs tab, then select a single Log. Below the generated response, find the Add correction button:
The Add correction button is below the Assistant response.
The corrected text and correction will appear next to one another, the correction is displayed in green.
Corrections are a great way to fine tune your models, to learn more, see Creating a Curated Dataset.