Capture user feedback on LLM responses

This cookbook covers how to implement structured feedback logging to continuously improve an FAQ chatbot’s accuracy, relevance, and user experience:

Capture real-time user ratings (good/bad) on chatbot responses
Log specific defects like grammatical errors, hallucinations, or ambiguity
Run an interactive chatbot loop to collect feedback in real time

Integrating feedback logging creates a chatbot that learns from user input and evolves over time.

Step 1: Install Dependencies

An Orq.ai account is required before starting. If not signed up yet, create an account first. A Google Colab file is also available to copy, replace the API key, and run immediately. Start by installing the required packages to use the Orq.ai SDK:

pip install orq-ai-sdk

Step 2: Identity Tracking (Optional)

Identities in Orq.ai help track user interactions and API usage across an application. They can represent users, teams, or projects and enable better analytics and budget management. Create an Identity through AI Studio:

Go to Identities in the workspace
Click Create an Identity
Add the identity details (name, email, externalId)
Set optional metadata and budget limits

To learn more about creating Identities, see Creating an Identity.

Step 3: Set Up the Orq Client

Set up the Orq.ai client using the API key. Replace the placeholder with the actual API key.

import os
from orq_ai_sdk import Orq

# Initialize Orq (Standalone Block for Initialization)
api_key = os.getenv("ORQ_API_KEY", "your_api_key_here")
client = Orq(
    api_key=api_key,
)
orq = client  # Maintain consistency for feedback logging

import { Orq } from '@orq-ai/node';

// Initialize Orq (Standalone Block for Initialization)
const apiKey = process.env.ORQ_API_KEY || 'your_api_key_here';
const client = new Orq({
  apiKey: apiKey,
});
const orq = client;

Step 4: Create a Knowledge Base

Create a knowledge base via the SDK, upload a document (replace docs.pdf with the product documentation file to be used by the FAQ bot), and attach it as a datasource. The agent created in the next step will automatically retrieve relevant chunks from this knowledge base at query time.

Python

import base64

def setup_knowledge_base():
    # Create the knowledge base
    kb = client.knowledge.create(request={
        "type": "internal",
        "key": "faq-docs",
        "embedding_model": "openai/text-embedding-3-small",
        "path": "CustomerSupport",
    })
    knowledge_id = kb.id

    # Upload a local file (base64-encoded)
    with open("docs.pdf", "rb") as f:
        encoded = base64.b64encode(f.read()).decode("utf-8")

    uploaded_file = client.files.create(
        filename="docs.pdf",
        content=encoded,
        content_type="application/pdf",
    )

    # Attach the file to the knowledge base as a datasource
    client.knowledge.create_datasource(
        knowledge_id=knowledge_id,
        file_id=uploaded_file.file.file_id,
        display_name="Product Docs",
    )

    print(f"Knowledge base ready: {knowledge_id}")
    return knowledge_id

knowledge_id = setup_knowledge_base()

For a more detailed explanation, see the Knowledge Base docs.

Step 5: Create the Agent

Create the FAQ agent via the SDK, attaching the knowledge base from Step 4. The agent instructions define the behavior: answer only from the knowledge base, express uncertainty when unsure.

Python

def setup_agent(knowledge_id):
    agent = client.agents.create(
        key="faq-bot",
        role="FAQ assistant",
        description="Answers user questions based on the product knowledge base",
        instructions="""Answer questions as accurately as possible using only the provided knowledge base.
If no relevant information is found, respond: "Sorry, I don't have information to answer that question."
Express uncertainty on unclear topics. Avoid speculation or personal opinions.""",
        path="CustomerSupport",
        model="openai/gpt-4o",
        settings={},
        knowledge_bases=[{"knowledge_id": knowledge_id}],
    )
    print(f"Agent ready: {agent.key}")
    return agent.key

agent_key = setup_agent(knowledge_id)

Step 6: Define the Interaction Function

The bot needs a function to send user messages to the agent and return the response text and trace ID. The trace ID is needed for feedback logging in Step 7. This function is called inside the chatbot loop in Step 7.

def chat_with_agent(message, agent_key):
    res = client.agents.responses.create(
        agent_key=agent_key,
        message={
            "role": "user",
            "parts": [{"kind": "text", "text": message}],
        },
        background=False,
        stream=False,
    )

    # Extract text from the last output message
    response = res.output[-1].parts[0].text

    # trace_id is used for feedback logging
    trace_id = res.telemetry.trace_id

    return response, trace_id

Step 7: Run the FAQ Bot

In a real deployment, feedback would be collected through front-end buttons (e.g., thumbs-up/down, dropdowns, or action buttons). For demonstration purposes, we simulate this process in the notebook using text-based inputs. The feedback loop works in two stages:

User Rating: After each response, users mark it as good or bad.
Defect Classification: For bad responses, users specify the issue (grammatical, hallucination, off-topic, etc.) for targeted improvements.

Both signals are logged as structured traces in Orq.ai, providing data to evaluate and iterate on the chatbot over time.

defect_options = [
    "grammatical", "spelling", "hallucination", "repetition", "inappropriate", "off_topic", "incompleteness", "ambiguity"
]

def chatbot(agent_key):
    print("\nChat started. Type 'exit' or 'quit' to end the chat.\n")

    while True:
        user_input = input("User: ")
        if user_input.lower() in ["exit", "quit"]:
            print("Ending chat.")
            break

        # Get model response
        response, trace_id = chat_with_agent(user_input, agent_key)
        print(f"Assistant: {response}")

        # Get feedback
        feedback = input("Provide feedback (good/bad) or press Enter to skip: ").strip().lower()
        if feedback in ["good", "bad"]:
            res = orq.feedback.create(request={"field": "rating", "value": [feedback], "trace_id": trace_id})
            print(f"Feedback logged: {res}")

            if feedback == "bad":
                # Log defect type
                defect_feedback = input("What was wrong with the response? (Choose from: grammatical, spelling, hallucination, repetition, inappropriate, off_topic, incompleteness, ambiguity): ").strip().lower()

                if defect_feedback in defect_options:
                    defect_res = orq.feedback.create(request={"field": "defects", "value": [defect_feedback], "trace_id": trace_id})
                    print(f"Defect feedback logged: {defect_res}")
                else:
                    print("Invalid defect type. No defect feedback logged.")

# Run chatbot
chatbot(agent_key)

Next Steps

A structured feedback loop is now in place for the FAQ bot, ensuring continuous learning and response improvement. To take it further:

Integrate interaction tracking: Link front-end actions (copied, saved, deleted, shared) to feedback logging, allowing the bot to learn without requiring explicit user input.
Create annotated datasets in Orq.ai: Use feedback as a selection method to build curated datasets for evaluation. Run experiments to see if updates to prompts, models, parameters, or the knowledge base improve performance and response quality.

Embedding feedback directly into user interactions creates a frictionless improvement cycle, making the FAQ bot more adaptive and user-friendly. For more resources and advanced features, visit the Orq.ai documentation.

Overview

Common Architectures

Chatbots & AI Apps

Data & Extraction

Evaluation & Safety

Integrations & Tooling

Learn

Capture user feedback on LLM responses

Step 1: Install Dependencies

Step 2: Identity Tracking (Optional)

Step 3: Set Up the Orq Client

Step 4: Create a Knowledge Base

Step 5: Create the Agent

Step 6: Define the Interaction Function

Step 7: Run the FAQ Bot

Next Steps

​Step 1: Install Dependencies

​Step 2: Identity Tracking (Optional)

​Step 3: Set Up the Orq Client

​Step 4: Create a Knowledge Base

​Step 5: Create the Agent

​Step 6: Define the Interaction Function

​Step 7: Run the FAQ Bot

​Next Steps

Step 1: Install Dependencies

Step 2: Identity Tracking (Optional)

Step 3: Set Up the Orq Client

Step 4: Create a Knowledge Base

Step 5: Create the Agent

Step 6: Define the Interaction Function

Step 7: Run the FAQ Bot

Next Steps