Skip to main content
This quick guide demonstrates how to build and evaluate an AI application for intent classification using the Orq platform. Before you begin, make sure you have an Orq account. If you don’t have one, sign up first. Additionally, we’ve prepared a Google Colab file that you can copy and run immediately, simplifying the setup process. Just replace the API key, and you’re ready to go! After completing this tutorial, visit the Orq documentation for more advanced topics Why Intent Classification is Useful Intent classification is a powerful tool for understanding and categorizing user inputs, making it essential for building smarter, more responsive applications. Here are a few use cases:
  • Customer Support: Automatically categorize support tickets or chatbot queries to route them to the right team.
  • E-commerce Personalization: Understand user intents like “buy,” “browse,” or “compare” to tailor shopping experiences.
  • Task Automation: Trigger specific workflows based on user commands, such as scheduling meetings or setting reminders.
Step 1: Install Dependencies Start by installing the required packages. These include the Orq SDK and additional libraries for handling datasets and managing environment variables. 
!pip install orq-ai-sdk datasets huggingface_hub
Step 2: Identity Tracking (Optional) Identities in Orq.ai help track user interactions and API usage across your application. They can represent users, teams, or projects and enable better analytics and budget management. Create an Identity through the Orq.ai Studio:
  1. Go to Identity Analytics in your workspace
  2. Click Create an Identity
  3. Add the user details (name, email, external ID)
  4. Set optional metadata and budget limits
Learn more about creating identities, see Creating an Identity.
Step 3: Initialize the Orq Client The Orq client allows you to communicate with the Orq platform. Set it up using your API key, which can be stored as an environment variable (ORQ_API_KEY) or passed directly. 
import os
from orq_ai_sdk import Orq

client = Orq(
    api_key=os.environ.get("ORQ_API_KEY", "your_orq_api_key"),
    identity_id="your_contact_id"  # Optional: add identity ID from Studio for tracking
)
Step 4: Load a Dataset Use the Hugging Face datasets library to load the dataset for intent classification. Here, we use a public dataset that contains user queries labeled with intents. 
from datasets import load_dataset

# Load dataset
dataset = load_dataset("Bhuvaneshwari/intent_classification")
df = dataset["train"].to_pandas()

​
Intent classification prompt

This deployment is designed to classify user inputs into specific intents to understand the purpose behind their requests. The model will identify the most appropriate intent from a predefined set, enabling precise and context-aware responses. Intent recognition is particularly useful when setting up chatbots This is the prompt in Orq.ai: 
You are tasked with identifying the intent behind user inputs based on the following intents:
'PlayMusic', 'AddToPlaylist', 'RateBook', 'Greetings', 'SearchScreeningEvent', 'BookRestaurant', 'GetWeather', 'Book Meeting', 'SearchCreativeWork', 'Cancellation', 'Affirmation', 'excitment'.

Here are some examples: 

Input: "Hey there, how are you doing?"
Intent: Greetings

Input: "Play the album Abbey Road by The Beatles."
Intent: PlayMusic

Input: "Add this song to my workout playlist."
Intent: AddToPlaylist

Input: "This book deserves a solid five-star rating."
Intent: RateBook

ONLY OUTPUT THE LABEL WITHOUT ''

here is the input that needs intent classification: {{text}}
Step 5: Invoke Orq Deployment Integrate the Orq intent classification model by invoking a deployed model for predictions. Iterate through the dataset and store the results in a new column called “output”. 
# Placeholder for deployment key
deployment_key = "intent_classification"

# Add predictions to the DataFrame
outputs = []
for _, row in df.iterrows():
    generation = client.deployments.invoke(
        key=deployment_key,
        context={"environments": []},
        inputs={"text": row["text"]},
        metadata={"custom-field-name": "custom-metadata-value"},
    )
    outputs.append(generation.choices[0].message.content)

# Add the output column
df["output"] = outputs
Step 6: Evaluate Model Performance Use metrics such as accuracy, precision, recall, and F1-score to assess the quality of your model’s predictions. 
from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

# Calculate metrics
accuracy = accuracy_score(df["intent"], df["output"])
precision = precision_score(df["intent"], df["output"], average="macro")
recall = recall_score(df["intent"], df["output"], average="macro")
f1 = f1_score(df["intent"], df["output"], average="macro")

# Display results
print(f"Accuracy: {accuracy:.4f}")
print(f"Precision: {precision:.4f}")
print(f"Recall: {recall:.4f}")
print(f"F1 Score: {f1:.4f}")
Feedback Collection (Optional) Feedback in Orq.ai helps track response quality and identify areas for improvement. You can collect user ratings, defect classifications, and corrections to continuously enhance your application. Provide feedback through the Orq.ai Studio:
  1. Go to Logs in your workspace
  2. Find the specific deployment invocation
  3. Use the feedback interface to rate responses
  4. Add defect classifications or corrections as needed
You can also collect feedback programmatically via the API if needed.
Next Steps Congratulations! You’ve successfully built and evaluated an intent classification application using Orq. To further enhance your application:
  • Explore other datasets and use cases.
  • Integrate the model into a chatbot or voice assistant.
  • Automate deployment and testing with Orq’s advanced features.
For more resources, visit the Orq documentation.