Human in the Loop feedback

Human in the Loop is a concept often used in artificial intelligence and large language models (LLMs) that involves humans in the decision-making or validation process when using language models, especially in situations where the model's predictions or actions may be uncertain, risky, or require human judgment and expertise.

This concept is also known as Reinforcement Learning from Human Feedback (RLHF).

RHLF is an approach that leverages human-provided feedback to train or fine-tune reinforcement learning models. This concept is particularly relevant in large language models (LLMs) because it helps improve their performance, safety, and alignment with human values.

This can be applied in several ways, including:

1. Data annotation and supervision: Since LLMs require substantial data for training, humans often curate and label datasets to ensure the model learns from high-quality, relevant information.
2. Fine-tuning: After pre-training on a large amount of text, LLMs can be fine-tuned for specific tasks or domains. Human experts can be involved in this fine-tuning process to adjust the model's behavior, making it more applicable to particular tasks or industries.
3. Evaluation and validation: LLMs can generate text, answer questions, or make recommendations, but the quality of these outputs can vary. Human reviewers can assess and validate the model's responses, ensuring they meet certain quality standards, are factually accurate, and align with ethical guidelines.
4. Content filtering and moderation: Human reviewers are employed to filter and moderate the model's output to prevent generating harmful or inappropriate content. This is crucial in applications like content generation, chatbots, and AI-driven customer support.

In orq.ai, there are several ways a human expert can help improve the performance of an LLM and guide its output or response.

Thumbs up and Thumbs down

This is a simple yet effective way for humans to provide feedback on the responses generated by a large language model (LLM). These feedback mechanisms are similar to the familiar concept of "liking" or "disliking" content on various online platforms, and they serve as a means of guiding the LLM's learning and improving its responses. Here's how these concepts work:

Thumbs Up (Positive Feedback):

When a user interacts with an LLM and receives a helpful, accurate, or satisfying response, they can provide a "thumbs up" or a positive rating to that response. This indicates that the LLM's output was valuable and aligned with the user's intent.

Thumbs Down (Negative Feedback):

Conversely, suppose a user receives a response from the LLM that is unhelpful, incorrect, offensive, or otherwise unsatisfactory. In that case, they can provide a "thumbs down" or a negative rating to that response. This feedback signals that the LLM's output needs improvement.

Using the add metrics method

Orq.ai provides users with the ability to use the addMetrics() method to add metadata, metrics, and information about the interaction with the LLM to the request log. This will help the human or domain expert add more custom metrics to the log.

For example: Add metrics to your request log.

deployment.addMetrics({
  feedback: {
    score: 70
  }
})

deployment.add_metrics(
  feedback={"score": 70}
)

Metadata

Metadata is a set of key-value pairs that you can use to add custom information to the log. It typically includes additional information or context related to the request or response, which can be helpful for various purposes. Here's how you can pass metadata to the addMetrics method:

deployment.addMetrics({
  chain_id: "c4a75b53-62fa-401b-8e97-493f3d299316",
  user_id: "e3a202a6-461b-447c-abe2-018ba4d04cd0",
  metadata: {
    custom: "custom_metadata",
    chain_id: "ad1231xsdaABw"
  }
})

deployment.add_metrics(
  chain_id="c4a75b53-62fa-401b-8e97-493f3d299316",
  user_id="e3a202a6-461b-447c-abe2-018ba4d04cd0",
  metadata={
      "custom": "custom_metadata",
      "chain_id": "ad1231xsdaABw",
  }
)

Review of response

A human expert can assess and evaluate the generated text produced by the model in response to a given prompt or query. This review process is crucial for assessing the quality, relevance, and safety of the LLM's output and ensuring it aligns with the intended purpose.

Advantages of Human in the Loop in LLMs

• Improved Quality and Trust: Human reviewers can help improve the quality and accuracy of LLM outputs, making them more reliable and trustworthy. This is particularly important in applications where errors or biases can have significant consequences.
• Adaptability and Customization: HITL allows LLMs to be customized for specific tasks or industries. Human input ensures the model aligns with domain-specific requirements and can handle nuances and complexities.
• Ethical Control: HITL can prevent generating harmful, biased, or inappropriate content by providing human oversight and moderation. This is essential for maintaining ethical standards.
• Addressing Uncertainty: LLMs can sometimes produce uncertain or ambiguous responses. Human reviewers can resolve such uncertainty, making the model more useful when clarity is essential.
• Continuous Learning: Human reviewers can provide feedback to LLMs, helping them learn from their mistakes and improve over time. This iterative feedback loop contributes to the ongoing refinement of the model.
• Compliance and Regulation: HITL can ensure that LLM outputs conform to legal and industry standards with strict guidelines in regulated industries or areas.

Wrap up

In essence, the human-in-the-loop approach emphasizes the importance of human feedback and metadata to improve the performance of AI systems. Several key elements contribute to the HITL process, including "Thumbs up" and "Thumbs down" feedback, the use of the "add metrics" method, metadata integration, and the review of AI-generated responses.