> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orq.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Capture user feedback on LLM responses

> Implement structured user feedback to improve an LLM chatbot. Capture ratings, log defects, and create a continuous learning loop for better AI responses.

This cookbook covers how to implement structured feedback logging to continuously improve an FAQ chatbot's accuracy, relevance, and user experience:

* Capture **real-time user ratings** (good/bad) on chatbot responses
* **Log specific defects** like grammatical errors, hallucinations, or ambiguity
* Run an interactive chatbot loop to collect feedback in real time

Integrating feedback logging creates a chatbot that learns from user input and evolves over time.

#### Step 1: Install Dependencies

An **Orq.ai** account is required before starting. If not signed up yet, [create an account first](https://orq.ai/create-account). A [Google Colab](https://colab.research.google.com/drive/1xvMyMwvNd6jKuf-6PVY5mqcg5rtYRHw6?usp=sharing) file is also available to copy, replace the API key, and run immediately.

Start by installing the required packages to use the **Orq.ai** SDK:

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
pip install orq-ai-sdk
```

#### Step 2: Identity Tracking (Optional)

Identities in **Orq.ai** help track user interactions and API usage across an application. They can represent users, teams, or projects and enable better analytics and budget management.

Create an Identity through **AI Studio**:

1. Go to **Identities** in the workspace
2. Click **Create an Identity**
3. Add the identity details (name, email, `externalId`)
4. Set optional metadata and budget limits

To learn more about creating Identities, see [Creating an Identity](/docs/ai-studio/observability/identities#creating-an-identity-using-the-api).

#### Step 3: Set Up the Orq Client

Set up the **Orq.ai** client using the API key. Replace the placeholder with the actual API key.

<CodeGroup>
  ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
  import os
  from orq_ai_sdk import Orq

  # Initialize Orq (Standalone Block for Initialization)
  api_key = os.getenv("ORQ_API_KEY", "your_api_key_here")
  client = Orq(
      api_key=api_key,
  )
  orq = client  # Maintain consistency for feedback logging
  ```

  ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
  import { Orq } from '@orq-ai/node';

  // Initialize Orq (Standalone Block for Initialization)
  const apiKey = process.env.ORQ_API_KEY || 'your_api_key_here';
  const client = new Orq({
    apiKey: apiKey,
  });
  const orq = client;
  ```
</CodeGroup>

#### Step 4: Create a Knowledge Base

Create a knowledge base via the SDK, upload a document (replace `docs.pdf` with the product documentation file to be used by the FAQ bot), and attach it as a datasource. The agent created in the next step will automatically retrieve relevant chunks from this knowledge base at query time.

```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import base64

def setup_knowledge_base():
    # Create the knowledge base
    kb = client.knowledge.create(request={
        "type": "internal",
        "key": "faq-docs",
        "embedding_model": "openai/text-embedding-3-small",
        "path": "CustomerSupport",
    })
    knowledge_id = kb.id

    # Upload a local file (base64-encoded)
    with open("docs.pdf", "rb") as f:
        encoded = base64.b64encode(f.read()).decode("utf-8")

    uploaded_file = client.files.create(
        filename="docs.pdf",
        content=encoded,
        content_type="application/pdf",
    )

    # Attach the file to the knowledge base as a datasource
    client.knowledge.create_datasource(
        knowledge_id=knowledge_id,
        file_id=uploaded_file.file.file_id,
        display_name="Product Docs",
    )

    print(f"Knowledge base ready: {knowledge_id}")
    return knowledge_id

knowledge_id = setup_knowledge_base()
```

For a more detailed explanation, see the [Knowledge Base docs](/docs/ai-studio/ai-engineering/knowledge-bases-memory-stores).

#### Step 5: Create the Agent

Create the FAQ agent via the SDK, attaching the knowledge base from Step 4. The agent instructions define the behavior: answer only from the knowledge base, express uncertainty when unsure.

```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
def setup_agent(knowledge_id):
    agent = client.agents.create(
        key="faq-bot",
        role="FAQ assistant",
        description="Answers user questions based on the product knowledge base",
        instructions="""Answer questions as accurately as possible using only the provided knowledge base.
If no relevant information is found, respond: "Sorry, I don't have information to answer that question."
Express uncertainty on unclear topics. Avoid speculation or personal opinions.""",
        path="CustomerSupport",
        model="openai/gpt-4o",
        settings={},
        knowledge_bases=[{"knowledge_id": knowledge_id}],
    )
    print(f"Agent ready: {agent.key}")
    return agent.key

agent_key = setup_agent(knowledge_id)
```

#### Step 6: Define the Interaction Function

The bot needs a function to send user messages to the agent and return the response text and trace ID. The trace ID is needed for feedback logging in Step 7. This function is called inside the chatbot loop in Step 7.

<CodeGroup>
  ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
  def chat_with_agent(message, agent_key):
      res = client.agents.responses.create(
          agent_key=agent_key,
          message={
              "role": "user",
              "parts": [{"kind": "text", "text": message}],
          },
          background=False,
          stream=False,
      )

      # Extract text from the last output message
      response = res.output[-1].parts[0].text

      # trace_id is used for feedback logging
      trace_id = res.telemetry.trace_id

      return response, trace_id
  ```
</CodeGroup>

#### Step 7: Run the FAQ Bot

In a real deployment, feedback would be collected through front-end buttons (e.g., thumbs-up/down, dropdowns, or action buttons). For demonstration purposes, we simulate this process in the notebook using text-based inputs.

The feedback loop works in two stages:

1. **User Rating**: After each response, users mark it as good or bad.
2. **Defect Classification**: For bad responses, users specify the issue (grammatical, hallucination, off-topic, etc.) for targeted improvements.

Both signals are logged as structured traces in **Orq.ai**, providing data to evaluate and iterate on the chatbot over time.

<CodeGroup>
  ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
  defect_options = [
      "grammatical", "spelling", "hallucination", "repetition", "inappropriate", "off_topic", "incompleteness", "ambiguity"
  ]

  def chatbot(agent_key):
      print("\nChat started. Type 'exit' or 'quit' to end the chat.\n")

      while True:
          user_input = input("User: ")
          if user_input.lower() in ["exit", "quit"]:
              print("Ending chat.")
              break

          # Get model response
          response, trace_id = chat_with_agent(user_input, agent_key)
          print(f"Assistant: {response}")

          # Get feedback
          feedback = input("Provide feedback (good/bad) or press Enter to skip: ").strip().lower()
          if feedback in ["good", "bad"]:
              res = orq.feedback.create(request={"field": "rating", "value": [feedback], "trace_id": trace_id})
              print(f"Feedback logged: {res}")

              if feedback == "bad":
                  # Log defect type
                  defect_feedback = input("What was wrong with the response? (Choose from: grammatical, spelling, hallucination, repetition, inappropriate, off_topic, incompleteness, ambiguity): ").strip().lower()

                  if defect_feedback in defect_options:
                      defect_res = orq.feedback.create(request={"field": "defects", "value": [defect_feedback], "trace_id": trace_id})
                      print(f"Defect feedback logged: {defect_res}")
                  else:
                      print("Invalid defect type. No defect feedback logged.")

  # Run chatbot
  chatbot(agent_key)
  ```
</CodeGroup>

#### Next Steps

A structured feedback loop is now in place for the FAQ bot, ensuring continuous learning and response improvement. To take it further:

* **Integrate interaction tracking**: Link front-end actions (copied, saved, deleted, shared) to feedback logging, allowing the bot to learn without requiring explicit user input.
* **Create annotated datasets in Orq.ai**: Use feedback as a selection method to build **curated datasets** for evaluation. Run experiments to see if updates to prompts, models, parameters, or the knowledge base improve performance and response quality.

Embedding feedback directly into user interactions creates a **frictionless improvement cycle**, making the FAQ bot more adaptive and user-friendly.

For more resources and advanced features, visit the **Orq.ai** documentation.