Advanced RAG

​

Objective

​

Use Case

• Multi-Source Intelligence: Search across different types of documentation and knowledge bases simultaneously (manuals, policies, historical data).
• Context-Aware Responses: Transform vague queries into precise searches and understand user intent across complex domains.
• Quality Validation: Ensure retrieved information is current, relevant, and authoritative through automated document grading.
• Confidence Scoring: Provide reliability indicators for responses to support critical decision-making.
• Continuous Learning: Track solution effectiveness and improve retrieval quality over time.
• Enterprise Support: Handle complex, multi-step scenarios requiring comprehensive knowledge synthesis.

​

Prerequisite

• Orq.ai Account: Active workspace in the Orq.ai Studio.
• API Access: Valid API key from Workspace Settings > API Keys
• Model Access: Multiple text generation models enabled in the Model Garden:
  • Primary model for response generation (e.g., gpt-4, claude-3-sonnet)
  • Secondary model for query refinement and document grading
• Embedding Model: At least one embedding model for knowledge base functionality (e.g., text-embedding-ada-002, text-embedding-3-large).
• Multiple Knowledge Sources: Diverse document types for comprehensive coverage:
  • Technical documentation (PDF, DOCX)
  • Product manuals and guides
• RAGAS Integration: Understanding of RAGAS evaluators for quality assessment (see RAGAS Evaluators).

​

Creating Multiple Knowledge Bases

​

1. Hardware Documentation Knowledge Base

• Choose a Project and Folder and select the + button.
• Choose Knowledge Base.
• Enter key: hardwareManuals and name: Hardware Documentation.
• Select an Embedding Model optimized for technical content.
• Upload hardware manuals (Dell, HP, Apple, network equipment guides).

​

2. Software Documentation Knowledge Base

• Create knowledge base with key: softwareGuides.
• Upload software documentation (Office 365, VPN clients, enterprise applications).
• Configure chunking for code examples and step-by-step procedures.

Multiple knowledge bases provide comprehensive coverage for complex queries.

​

Enabling Agentic RAG

• Navigate to each Knowledge Base Settings.
• Toggle on Agentic RAG.
• Select a model for query refinement and document grading.
• Configure grading strictness based on information criticality.

Agentic RAG enables intelligent query refinement and document validation.

​

Configuring Advanced RAG Deployment

• Choose a Project and Folder and select the + button.
• Choose Deployment.
• Enter name advancedITAssistant.
• Choose a primary Model (e.g., gpt-4).

You are an advanced IT support assistant that provides comprehensive, step-by-step troubleshooting guidance by intelligently searching across multiple knowledge sources.

Your capabilities include:
- Multi-source information synthesis from hardware manuals, software guides, IT policies, and historical solutions
- Context-aware query understanding and refinement
- Solution validation and confidence scoring
- Step-by-step guidance with fallback strategies

Knowledge Sources Available:
- Hardware Documentation: {{hardwareManuals}} - For hardware-related issues and specifications
- Software Guides: {{softwareGuides}} - For application and software troubleshooting

Instructions:
1. Analyze the user's issue to understand the problem domain (hardware, software, network, security)
2. Search relevant knowledge bases based on the issue type
3. Synthesize information from multiple sources when needed
4. Provide step-by-step solutions with clear confidence indicators
5. Include source attribution and policy compliance notes
6. Offer alternative solutions when primary approach may not work
7. If solution requires escalation, clearly explain why and to whom

Response Format:
- Start with issue classification and confidence level
- Provide primary solution with numbered steps
- Add alternative approaches if applicable
- Note any policy considerations or restrictions
- Suggest escalation path if needed

Always prioritize user safety and company policy compliance.

​

Adding Multiple Knowledge Bases

• Open the Knowledge Base tab in the Configuration screen.
• Add each knowledge base with appropriate triggers:
  • Hardware Manuals: Keywords like “laptop”, “printer”, “network”, “hardware”
  • Software Guides: Keywords like “email”, “VPN”, “application”, “software”

Multiple knowledge bases configured with intelligent triggering.

​

Adding Quality Evaluation

• Open the Evaluators tab in the Settings screen
• First make sure an evaluator is available within your Project by creating it (see RAGAS Evaluators).
• Click Add Evaluator:
  • RAGAS Response Relevancy: Verify answers address the question
  • RAGAS Coherence: Check response structure and flow
• Set their sample rate to define how often will the evaluators be run.

​

Integrating with the SDK

pip install orq-ai-sdk

import os
from orq_ai_sdk import Orq

client = Orq(
    api_key=os.environ.get("ORQ_API_KEY", "__API_KEY__"),
    environment="production",
    contact_id="it_assistant" # optional
)

import os
from orq_ai_sdk import Orq

class ITAssistant:
    def __init__(self, client: Orq):
        self.client = client
        self.deployment_key = "advancedITAssistant"
    
    def ask_question(self, issue: str):
        """Ask the IT assistant a question"""
        
        # Call the deployment
        response = self.client.deployments.invoke(
            key=self.deployment_key,
            messages=[{"role": "user", "content": issue}],
            context={"include_retrievals": True}
        )
        
        return {
            "answer": response.choices[0].message.content,
            "sources": getattr(response, 'retrievals', [])
        }

# Initialize the assistant
client = Orq(
    api_key=os.environ.get("ORQ_API_KEY", "__API_KEY__"),
    environment="production"
)

assistant = ITAssistant(client)

# Example usage
result = assistant.ask_question(
    "My laptop won't connect to the corporate WiFi after the latest macOS update, and I can't access my email or VPN."
)

print("Answer:", result["answer"])

❯ python3 it_assistant.py
Answer: Issue Classification: Network connection and application access failure after macOS update
Confidence Level: High

Primary Solution:

1. Verify your laptop's wireless adapter is enabled and functioning properly:
   - Go to System Preferences > Network and check that the wireless adapter is listed and "Connected" to the network.
   - If the adapter is disabled, click the "Turn Wi-Fi On" button.

2. Check your corporate WiFi network settings:
   - Ensure you are attempting to connect to the correct SSID (network name) for your office.
   - Confirm the network security settings (e.g. WPA2, 802.1X) match what your IT team has provided.
   - If you have previously connected successfully, the network details should be pre-configured.

3. Reset your network settings:
   - Go to System Preferences > Network
   - Click the "Advanced" button, then the "Renew DHCP Lease" option.
   - If that doesn't work, try deleting the existing WiFi network and re-adding it.

​

Viewing Logs and Analytics

• Multi-Source Retrievals: See which knowledge bases were accessed and what documents were retrieved from each source
• Query Enhancement: View how Agentic RAG refined the original query for better retrieval
• Quality Metrics: RAGAS evaluation scores automatically computed by the evaluators configured in your deployment
• Source Attribution: Detailed breakdown of which sources contributed to the final response
• Performance Metrics: Response times for each stage of the advanced RAG pipeline

Evaluators will be seen within the Feedback window of the logs.