Advanced RAG with multi-source retrieval

Objective

An Advanced RAG (Retrieval-Augmented Generation) system provides sophisticated multi-source information retrieval with intelligent query enhancement, document validation, and quality assessment. This architecture demonstrates how to build enterprise-grade AI assistants that can intelligently orchestrate across multiple knowledge sources.

Use Case

Advanced RAG is ideal for applications that need:

Multi-Source Intelligence: Search across different types of documentation and knowledge bases simultaneously (manuals, policies, historical data).
Context-Aware Responses: Transform vague queries into precise searches and understand user intent across complex domains.
Quality Validation: Ensure retrieved information is current, relevant, and authoritative through automated document grading.
Confidence Scoring: Provide reliability indicators for responses to support critical decision-making.
Continuous Learning: Track solution effectiveness and improve retrieval quality over time.
Enterprise Support: Handle complex, multi-step scenarios requiring comprehensive knowledge synthesis.

In this example, we’ll create an enterprise support bot that sources helpdesk data from multiple knowledge bases.

Prerequisite

Before configuring an Advanced RAG system, ensure you have:

Orq.ai Account: Active workspace in the AI Studio.
API Access: Valid API key from Workspace Settings > API Keys
Model Access: Multiple text generation models enabled in the AI Gateway:
- Primary model for response generation (e.g., gpt-4, claude-sonnet-4-6)
- Secondary model for query refinement and document grading
Embedding Model: At least one embedding model for knowledge base functionality (e.g., text-embedding-ada-002, text-embedding-3-large).
Multiple Knowledge Sources: Diverse document types for comprehensive coverage:
- Technical documentation (PDF, DOCX)
- Product manuals and guides
RAGAS Integration: Understanding of RAGAS evaluators for quality assessment (see RAGAS Evaluators).

Creating Multiple Knowledge Bases

Advanced RAG requires specialized knowledge bases for different information types. Create separate knowledge bases for optimal retrieval performance:

1. Hardware Documentation Knowledge Base

Choose a Project and Folder and select the button.
Choose Knowledge Base.
Enter key: hardwareManuals and name: Hardware Documentation.
Select an Embedding Model optimized for technical content.
Upload hardware manuals (Dell, HP, Apple, network equipment guides).

2. Software Documentation Knowledge Base

Create knowledge base with key: softwareGuides.
Upload software documentation (Office 365, VPN clients, enterprise applications).
Configure chunking for code examples and step-by-step procedures.

Multiple knowledge bases provide comprehensive coverage for complex queries.

Enabling Agentic RAG

For each knowledge base, enable Agentic RAG to improve retrieval quality:

Navigate to each Knowledge Base Settings.
Toggle on Agentic RAG.
Select a model for query refinement and document grading.
Configure grading strictness based on information criticality.

Agentic RAG enables intelligent query refinement and document validation.

Configuring Advanced RAG Deployment

Create a sophisticated deployment that orchestrates multiple knowledge sources:

Choose a Project and Folder and select the button.
Choose Deployment.
Enter name advancedITAssistant.
Choose a primary Model (e.g., gpt-4).

Configure the system message for intelligent multi-source orchestration:

YAML

You are an advanced IT support assistant that provides comprehensive, step-by-step troubleshooting guidance by intelligently searching across multiple knowledge sources.

Your capabilities include:
- Multi-source information synthesis from hardware manuals, software guides, IT policies, and historical solutions
- Context-aware query understanding and refinement
- Solution validation and confidence scoring
- Step-by-step guidance with fallback strategies

Knowledge Sources Available:
- Hardware Documentation: {{hardwareManuals}} - For hardware-related issues and specifications
- Software Guides: {{softwareGuides}} - For application and software troubleshooting

Instructions:
1. Analyze the user's issue to understand the problem domain (hardware, software, network, security)
2. Search relevant knowledge bases based on the issue type
3. Synthesize information from multiple sources when needed
4. Provide step-by-step solutions with clear confidence indicators
5. Include source attribution and policy compliance notes
6. Offer alternative solutions when primary approach may not work
7. If solution requires escalation, clearly explain why and to whom

Response Format:
- Start with issue classification and confidence level
- Provide primary solution with numbered steps
- Add alternative approaches if applicable
- Note any policy considerations or restrictions
- Suggest escalation path if needed

Always prioritize user safety and company policy compliance.

Adding Multiple Knowledge Bases

Open the Knowledge Base tab in the Configuration screen.
Add each knowledge base with appropriate triggers:
- Hardware Manuals: Keywords like “laptop”, “printer”, “network”, “hardware”
- Software Guides: Keywords like “email”, “VPN”, “application”, “software”

Multiple knowledge bases configured with intelligent triggering.

Adding Quality Evaluation

To enable automatic quality assessment with RAGAS evaluators:

Open the Evaluators tab in the Settings screen
First make sure an evaluator is available within your Project by creating it (see RAGAS Evaluators).
Click Add Evaluator:
- RAGAS Response Relevancy: Verify answers address the question
- RAGAS Coherence: Check response structure and flow
Set their sample rate to define how often will the evaluators be run.

The evaluators will automatically run on each generation and provide quality scores in the logs.

You can decide to block queries that don’t pass a certain threshold in evaluations, to do so, see Evaluators & Guardrails.

Learn more about Agentic RAG in Agentic RAG, knowledge base configuration in Knowledge Base, and RAGAS evaluators in RAGAS Evaluators.

When ready with your Deployment choose Deploy, learn more about Deployment Versioning.

Integrating with the SDK

Choose your preferred programming language and install the corresponding SDK:

pip install orq-ai-sdk

npm install @orq-ai/node

Get your integration ready by initializing the SDK as follows:

import os
from orq_ai_sdk import Orq

client = Orq(
    api_key=os.environ.get("ORQ_API_KEY", "__API_KEY__"),
)

import { Orq } from "@orq-ai/node";

const client = new Orq({
    apiKey: process.env.ORQ_API_KEY || "__API_KEY__",
});

To implement a simple Advanced RAG IT Assistant:

import os
from orq_ai_sdk import Orq

class ITAssistant:
    def __init__(self, client: Orq):
        self.client = client
        self.deployment_key = "advancedITAssistant"
    
    def ask_question(self, issue: str):
        """Ask the IT assistant a question"""
        
        # Call the deployment
        response = self.client.deployments.invoke(
            key=self.deployment_key,
            messages=[{"role": "user", "content": issue}],
            context={"include_retrievals": True}
        )
        
        return {
            "answer": response.choices[0].message.content,
            "sources": getattr(response, 'retrievals', [])
        }

# Initialize the assistant
client = Orq(
    api_key=os.environ.get("ORQ_API_KEY", "__API_KEY__"),
)

assistant = ITAssistant(client)

# Example usage
result = assistant.ask_question(
    "My laptop won't connect to the corporate WiFi after the latest macOS update, and I can't access my email or VPN."
)

print("Answer:", result["answer"])

import { Orq } from "@orq-ai/node";

class ITAssistant {
    constructor(client) {
        this.client = client;
        this.deploymentKey = "advancedITAssistant";
    }
    
    async askQuestion(issue) {
        // Call the deployment
        const response = await this.client.deployments.invoke({
            key: this.deploymentKey,
            messages: [{ role: "user", content: issue }],
            context: { include_retrievals: true }
        });
        
        return {
            answer: response.choices[0].message.content,
            sources: response.retrievals || []
        };
    }
}

// Initialize the assistant
const client = new Orq({
    apiKey: process.env.ORQ_API_KEY || "__API_KEY__",
});

const assistant = new ITAssistant(client);

// Example usage
const result = await assistant.askQuestion(
    "My laptop won't connect to the corporate WiFi after the latest macOS update, and I can't access my email or VPN."
);

console.log("Answer:", result.answer);
console.log(`Sources used: ${result.sources.length} documents`);

Here is what the output looks like:

❯ python3 it_assistant.py
Answer: Issue Classification: Network connection and application access failure after macOS update
Confidence Level: High

Primary Solution:

1. Verify your laptop's wireless adapter is enabled and functioning properly:
   - Go to System Preferences > Network and check that the wireless adapter is listed and "Connected" to the network.
   - If the adapter is disabled, click the "Turn Wi-Fi On" button.

2. Check your corporate WiFi network settings:
   - Ensure you are attempting to connect to the correct SSID (network name) for your office.
   - Confirm the network security settings (e.g. WPA2, 802.1X) match what your IT team has provided.
   - If you have previously connected successfully, the network details should be pre-configured.

3. Reset your network settings:
   - Go to System Preferences > Network
   - Click the "Advanced" button, then the "Renew DHCP Lease" option.
   - If that doesn't work, try deleting the existing WiFi network and re-adding it.

Viewing Logs and Analytics

Going back to the Deployment page, you can view the calls made through your Advanced RAG IT Assistant. The logging provides comprehensive insights into the multi-source retrieval process and quality validation. You can view details for a single log by clicking on a log line. This opens a panel containing all the details for the log, including:

Multi-Source Retrievals: See which knowledge bases were accessed and what documents were retrieved from each source
Query Enhancement: View how Agentic RAG refined the original query for better retrieval
Quality Metrics: RAGAS evaluation scores automatically computed by the evaluators configured in your deployment
Source Attribution: Detailed breakdown of which sources contributed to the final response
Performance Metrics: Response times for each stage of the advanced RAG pipeline

Evaluators will be seen within the Feedback window of the logs.

Congratulations for creating a complex RAG deployment with multiple Knowledge Bases, RAGAS Evaluators and integration.

Overview

Common Architectures

Chatbots & AI Apps

Data & Extraction

Evaluation & Safety

Integrations & Tooling

Learn

Advanced RAG with multi-source retrieval

Objective

Use Case

Prerequisite

Creating Multiple Knowledge Bases

1. Hardware Documentation Knowledge Base

2. Software Documentation Knowledge Base

Enabling Agentic RAG

Configuring Advanced RAG Deployment

Adding Multiple Knowledge Bases

Adding Quality Evaluation

Integrating with the SDK

Viewing Logs and Analytics

​Objective

​Use Case

​Prerequisite

​Creating Multiple Knowledge Bases

​1. Hardware Documentation Knowledge Base

​2. Software Documentation Knowledge Base

​Enabling Agentic RAG

​Configuring Advanced RAG Deployment

​Adding Multiple Knowledge Bases

​Adding Quality Evaluation

​Integrating with the SDK

​Viewing Logs and Analytics

Objective

Use Case

Prerequisite

Creating Multiple Knowledge Bases

1. Hardware Documentation Knowledge Base

2. Software Documentation Knowledge Base

Enabling Agentic RAG

Configuring Advanced RAG Deployment

Adding Multiple Knowledge Bases

Adding Quality Evaluation

Integrating with the SDK

Viewing Logs and Analytics