​

Home Dashboard Revamp

• Quick actions to get started right away. Create a deployment, run an evaluation, open the playground, or start an experiment directly from the dashboard. These workflows used to live in the sidebar, but we have brought them front and center to make your next step obvious and effortless.
• A redesigned layout with central dashboards and overview tables for a clear snapshot of your workspace performance.
• A Recently Viewed section so you can quickly jump back into your most recent experiments, deployments, or files as your workspace grows.
• Detailed model and deployment insights that help you understand which models drive requests, cost, and latency.

​

Trace Automation

• Advanced filtering across any trace attribute such as metadata, model, provider, cost, tokens, or project. Combine multiple filters and apply sampling to focus only on the traces that matter.
• Add to Dataset action that lets you automatically collect traces that match your filters. These datasets stay up to date as new traces arrive, so you can run experiments on the latest production data at any time.

​

Model Garden Revamp

• Multimodality visibility clearly shows which models support text, image, or both for input and output.
• Intelligence and speed ratings help you compare models at a glance based on performance and capability.
• Comprehensive token and pricing data display input and output costs to make budgeting and optimization simple.
• Filter and sort controls across provider, model type, and activity status make model discovery effortless.
• Gateway code snippets are available directly from each model card, letting you copy and deploy instantly through the Orq AI Gateway.

​

Mistral and Custom ElevenLabs Integration

• Mistral and ElevenLabs added as new providers with Bring Your Own Key (BYOK) support

​

Private OpenAI-Compatible Model Integration

• Custom model configuration for any OpenAI-compatible endpoint, including chat, completion, embedding, or image models.
• Bring your own API key and base URL to connect private or hosted models securely through the Orq AI Gateway.
• Full parameter control including temperature, max tokens, input and output token pricing, and region.
• Add detailed model description for easy discovery and team collaboration.

• Dataset creation from real production traces
• A/B testing between different models or configurations
• Evaluator workflows to continuously measure quality and outcomes

​

Experiment Comparison View Update

• Redesigned layout that moves the input variables and expected output to the left, giving you a cleaner overview when comparing multiple models or longer text outputs.
• Evaluator section has been moved to the bottom of the comparison view, keeping the focus on model responses while maintaining quick access to evaluation scores and feedback.
• Optimized for clarity and readability, especially when working with large datasets or extended completions.

​

Structured Outputs

• Improved JSON editors and clearer validation

​

UI and UX Improvements

• Enhanced error handling across prompts, tools, and experiments
• Updated icons, spacing, and alignment
• More responsive layouts across Playgrounds, Deployments, and Evaluators

​

Custom Reports

• Customizable views with filters for time range and project selection, allowing you to focus on the data that matters most.
• Build your own custom report by selecting and combining charts from different clusters.
• Comprehensive insights with prebuilt charts for:
  • Cost breakdowns and model-provider splits
  • Error rates and anomaly detection
  • Response time by model and latency analysis
  • Request and token volume tracking
  

​

Claude Haiku 4.5 now Available

​

n8n Integration now Available

​

What’s New

• Seamless Deployment Integration: Invoke any Orq Deployment directly from your n8n workflows
• Knowledge Base Search: Search through Knowledge Bases and retrieve relevant content.
• 1,000+ App Connections: Connect your LLM applications with popular tools including databases, APIs, CRMs, messaging platforms, and more.

​

Use Cases

• Automate content generation workflows across multiple platforms.
• Build intelligent data processing pipelines with RAG-enabled Deployments.
• Orchestrate complex multi-step AI workflows with conditional logic.
• Integrate LLM outputs with CRMs, databases, and notification systems.

​

Getting Started

1. Install the Orq Deployment node in your n8n instance (requires owner permissions).
2. Configure authentication with your Orq.ai API Key.
3. Add a Orq Deployment node or Knowledge Base node to your n8n workflow.
4. Connect with other n8n nodes to build complete automations

​

Tools API

​

What’s new

• HTTP Tools – External API integrations with templating & authentication.
  • URL templates with {{parameter}} substitution
  • Support for HTTP methods (GET, POST, PUT, DELETE, etc.)
  • Custom headers (including authentication)
  • Parameter definitions with types, descriptions, default values
  • Control over which parameters are visible to the LLM (send_to_model)
• Function Tools – Define tool schemas via API (same as in the UI).
  • OpenAPI-style schemas for custom business logic
  • Function name, description, parameters
  • Strict parameter validation
  • Optional manual approval before execution
• Code Tools (Python) – Run custom Python code in a secure sandbox.
  • Define parameters in JSON Schema
  • Access params variable for LLM-provided inputs
  • Supports return value processing & error handling
  • Fully isolated execution environment

​

Memory Stores API

​

Key Features

​

Persistent & Isolated Memory

• Entity-based Isolation: Use entity_id to scope memory per user, session, or organization.
• Multi-Tenant Support: Secure separation of data across entities.

​

Structured Memory Storage

• Memory Stores: Named containers for storing different types of contextual data.
• Built-in Tools: Automatic read, write, and query operations for seamless integration.

​

Agent Memory Workflow

• Discover First: Agents must use retrieve_memory_stores to discover available memory before interacting with them.
• Query Smartly: Once discovered, use query_memory_store to fetch relevant information before responding.
• System Prompt Guidance: Example:

  “Use retrieve_memory_stores to see what memory stores are available, then use query_memory_store to search for relevant information before responding.”

​

Streaming & Entity Tracking

• Streaming Support: Use entity_id for real-time memory tracking during ongoing conversations.

​

Improved Experiments

​

Claude Sonnet 4.5 now Available

​

Agents - Now in BETA

​

Key Features

​

Built on Modern Standards

​

Intelligent Memory & Context

• Memory Stores: Persistent context across conversations with entity-based isolation.
• Knowledge Base: Ground responses in your data with dynamic discovery and querying

​

Multi-Modal Processing

• File Processing - Handle images and PDFs with support varying per given model
• Search & Web Scraping - Bring in real-time insights with built-in search and scraping tools

​

Flexible Tool Ecosystem

• HTTP Tools - Seamless external API integrations with parameter templating and authentication
• Function Tools - Custom business logic and access to internal systems with OpenAI function calling schema validation
• Code Tools - Custom Python logic and transformations in secure sandbox environments

​

Architecture

• Multi-Agent Systems - Build sophisticated hierarchies with specialized sub-agents and intelligent delegation.
• Synchronous and streaming endpoints for different use cases.

​

Example

curl --location 'https://my.orq.ai/v2/agents/stream-run' \
--header 'Content-Type: application/json' \
--header 'Authorization: $ORQ_API_KEY' \
--data '{
    "key": "simple-agent-1",
    "role": "Assistant",
    "description": "A helpful assistant for general tasks",
    "instructions": "Be helpful and concise",
    "settings": {
        "max_iterations": 3,
        "max_execution_time": 300,
        "tools": [
            {
                "type": "current_date"
            }
        ]
    },
    "message": {
        "role": "user",
        "parts": [
            {
                "kind": "text",
                "text": "What'\''s today'\''s date and day of the week?"
            }
        ]
    },
    "model": "openai/gpt-4o",
    "path": "Default/agents",
    "memory_stores": [],
    "team_of_agents": []
}'

​

Release 3.12

​

🔍 Enhanced Observability & Debugging

​

Human-in-the-Loop Reviews

• Collect structured feedback on AI outputs with customizable Human Review Sets per trace type
• Directly add spans to datasets for continuous improvement
• Track contact IDs and thread context across chat completions

​

Faster Root Cause Analysis

• View retrieval configurations directly in span properties
• See evaluator names on spans for quick performance assessment
• Expanded OpenTelemetry support for more frameworks

​

Cost Optimization

• Optional response caching to reduce latency and API costs
• Fixed cost aggregation for image operations and Azure OpenAI
• More accurate token and billing tracking

​

🧪 Streamlined Experimentation

​

Improved Experiment Management

• Search across experiment entries
• Protection against accidental re-runs
• Persistent column settings and better cancellation handling
• Enhanced UI with clearer active states and progress indicators

​

🚀 AI Gateway Enhancements

​

Advanced Request Handling

• Automatic retries and fallback models for improved reliability
• Thread and contact tracking for conversation continuity
• Specify prompt versions directly in LLM calls
• Improved SSE streaming performance

​

💰 Budget Controls

​

Workspace-Level Cost Management

• Set and monitor budgets at workspace and contact levels
• New Budgets API for programmatic cost control
• Automated alerts and spending limits

​

🎯 Platform Improvements

​

Model Management

• New image generation models and providers
• Intelligent model filtering based on capabilities
• Improved cost extraction and model selection UI

​

Developer Experience

• Better API parameter documentation for Knowledge Base
• Unsaved changes protection across Teams and Contacts
• Improved error handling and retry logic throughout

​

AI Gateway - Superpower your LLM requests

• Fallbacks
• Retry
• Contact Tracking
• Thread Management
• Cache
• Knowledge Bases

curl --location 'https://api.orq.ai/v2/router/chat/completions' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $ORQ_API_KEY' \
--data-raw '{
  "model": "openai/gpt-4o",
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful customer support agent for {{company_name}}. Use available knowledge to assist {{customer_tier}} customers."
    },
    {
      "role": "user",
      "content": "I need help with API integration for my {{use_case}} project"
    }
  ],
  "orq": {
    "retry": {
      "count": 3,
      "on_codes": [429, 500, 502, 503, 504]
    },
    "fallbacks": [
      {
        "model": "anthropic/claude-3-5-sonnet-20241022"
      },
      {
        "model": "openai/gpt-4o-mini"
      }
    ],
    "cache": {
      "type": "exact_match",
      "ttl": 1800
    },
    "knowledge_bases": [
      {
        "knowledge_id": "api-documentation",
        "top_k": 5,
        "threshold": 0.75
      },
      {
        "knowledge_id": "integration-examples",
        "top_k": 3,
        "threshold": 0.8
      }
    ],
    "contact": {
      "id": "enterprise_customer_001",
      "display_name": "Enterprise User",
      "email": "user@enterprise.com"
    },
    "thread": {
      "id": "support_session_001",
      "tags": ["api-integration", "enterprise", "technical-support"]
    },
    "inputs": {
      "company_name": "Orq AI",
      "customer_tier": "Enterprise",
      "use_case": "e-commerce platform"
    }
  }
}'

​

OpenTelemetry - LangChain, LangGraph, OpenAI Agents and more

• Complete request tracing across LLM calls, chain executions, and agent workflows
• Automatic instrumentation for latency, token usage, and error tracking
• Zero-code instrumentation for supported frameworks

• Identify bottlenecks in complex multi-step AI workflows
• Track costs with token-level granularity
• Debug agent reasoning paths and tool usage in production
• Correlate AI operations with upstream/downstream services

​

Supported frameworks

• Agno
• AutoGen
• BeeAI
• CrewAI
• DSPy
• Google ADK
• Haystack
• Instructor
• LangChain / LangGraph
• LiteLLM
• LiveKit
• LlamaIndex
• Mastra
• OpenAI Agents
• Pydantic AI
• Semantic Kernel
• SmolAgents
• Vercel AI SDK

​

AI Gateway - Vercel AI SDK

• Full Vercel AI SDK Compatibility: Works with all Vercel AI SDK functions (generateText, streamText, embed, etc.)
• Multiple Model Types: Support for chat, completion, embedding, and image generation models
• Streaming Support: Real-time streaming responses for a better user experience
• Type-safe: Fully written in TypeScript with comprehensive type definitions
• Orq Platform Integration: Direct access to Orq AI’s model routing and optimization

​

Installation

npm i ai @orq-ai/vercel-provider

​

Getting Started

import { createOrqAiProvider } from "@orq-ai/vercel-provider";
import { generateText } from "ai";

const orq = createOrqAiProvider({
  apiKey: process.env.ORQ_API_KEY,
});

const { text } = await generateText({
  model: orq("gpt-4"),
  messages: [{ role: "user", content: "Hello!" }],
});

​

Human Reviews in Traces

​

Custom instrumentation with @traced

• Automatically logs function inputs, outputs, and metadata
• Supports nested spans and custom span types (LLM, agent, tool, etc.)
• Works seamlessly with the Orq SDK initialization (no separate init required)
• Integrates with OpenTelemetry for end-to-end distributed tracing

import time

@traced
def process_user(user_id: str, action: str) -> dict:
    print('process user')
    """Process user data with custom attributes."""
    # Simulate some processing
    time.sleep(0.1)

    result = {
        "user_id": user_id,
        "action": action,
        "status": "completed",
        "timestamp": time.time()
    }

    return result

pip install orq-ai-sdk

​

Cerebas and Jina support in the Model Garden

​

ByteDance Image Models

​

Examples

curl https://api.orq.ai/v2/proxy/images/generations \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -d '{
    "model": "bytedance/seedream-3-0-t2i-250415",
    "prompt": "A beautiful sunset over mountains",
    "n": 1
  }'

from openai import OpenAI
import os

client = OpenAI(
  base_url="https://api.orq.ai/v2/proxy",
  api_key=os.getenv("ORQ_API_KEY"),
)

response = client.images.generate(
  model="bytedance/seedream-3-0-t2i-250415",
  prompt="A beautiful sunset over mountains",
  n=1
)

print(response.data[0].url)

import OpenAI from "openai";

const openai = new OpenAI({
  baseURL: 'https://api.orq.ai/v2/proxy',
  apiKey: process.env.ORQ_API_KEY,
});

async function main() {
  const response = await openai.images.generate({
    model: "bytedance/seedream-3-0-t2i-250415",
    prompt: "A beautiful sunset over mountains",
    n: 1
  });

  console.log(response.data[0].url);
}

main();

curl https://api.orq.ai/v2/proxy/images/generations \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -d '{
    "model": "bytedance/seededit-3-0-i2i-250628",
    "prompt": "A beautiful sunset over mountains",
    "n": 1
  }'

from openai import OpenAI
import os

client = OpenAI(
  base_url="https://api.orq.ai/v2/proxy",
  api_key=os.getenv("ORQ_API_KEY"),
)

response = client.images.generate(
  model="bytedance/seededit-3-0-i2i-250628",
  prompt="A beautiful sunset over mountains",
  n=1
)

print(response.data[0].url)

import OpenAI from "openai";

const openai = new OpenAI({
  baseURL: 'https://api.orq.ai/v2/proxy',
  apiKey: process.env.ORQ_API_KEY,
});

async function main() {
  const response = await openai.images.generate({
    model: "bytedance/seededit-3-0-i2i-250628",
    prompt: "A beautiful sunset over mountains",
    n: 1
  });

  console.log(response.data[0].url);
}

main();

​

Workspace Budget

​

Introducing Evaluatorq

• Parallel Execution: Run multiple evaluation jobs concurrently with progress tracking
• Flexible Data Sources: Support for inline data, promises, and Orq platform datasets
• Type-safe: Fully written in TypeScript
• Orq Platform Integration: Seamlessly fetch and evaluate datasets from Orq AI (optional)

import { evaluatorq, job } from "@orq-ai/evaluatorq";

const textAnalyzer = job("text-analyzer", async (data) => {
  const text = data.inputs.text;
  const analysis = {
    length: text.length,
    wordCount: text.split(" ").length,
    uppercase: text.toUpperCase(),
  };

  return analysis;
});

await evaluatorq("text-analysis", {
  data: [
    { inputs: { text: "Hello world" } },
    { inputs: { text: "Testing evaluation" } },
  ],
  jobs: [textAnalyzer],
  evaluators: [
    {
      name: "length-check",
      scorer: async ({ output }) => {
        const passesCheck = output.length > 10;
        return {
          value: passesCheck ? 1 : 0,
          explanation: passesCheck
            ? "Output length is sufficient"
            : `Output too short (${output.length} chars, need >10)`,
        };
      },
    },
  ],
});

​

Native support for Gemini models - URL Context Tool

• Two-step retrieval: internal index cache first, live fetch if unavailable.
• Returns url_context_metadata with retrieval details and safety checks.
• Supports up to 20 URLs per request, max 34MB per URL.
• Retrieved content counts toward input tokens (tracked in usage_metadata).

• gemini-2.5-pro
• gemini-2.5-flash
• gemini-2.5-flash-lite

• Use direct, specific, fully qualified URLs (https://…).
• Avoid login-restricted or paywalled content.
• Combine with Google Search grounding for broader workflows.

​

OpenTelemetry Tracing Support

• Native OpenTelemetry (OTel) export for traces.
• Send spans directly to our OTLP/HTTP collector at: https://api.orq.ai/v2/otel.
• Works with standard OTel SDKs and exporters (Python, Node.js).
• Supports custom headers (e.g., bearer tokens) and resource attributes (e.g., service.name).

# Common env vars (both Python & Node.js)
export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.orq.ai/v2/otel"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <ORQ_API_KEY>"
export OTEL_RESOURCE_ATTRIBUTES="service.name=my-service,service.version=1.0.0"

# pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http opentelemetry-instrumentation

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

resource = Resource.create({
    "service.name": "my-python-service",
    "service.version": "1.0.0"
})

provider = TracerProvider(resource=resource)
trace.set_tracer_provider(provider)

exporter = OTLPSpanExporter(
    endpoint="https://api.orq.ai/v2/otel",
    # Optional if your setup requires auth
    headers={"Authorization": "Bearer <ORQ_API_KEY>"}
)

provider.add_span_processor(BatchSpanProcessor(exporter))

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("demo-span"):
    # your app logic here
    pass

import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { Resource } from '@opentelemetry/resources';

const traceExporter = new OTLPTraceExporter({
  url: 'https://api.orq.ai/v2/otel',
  // Optional if your setup requires auth
  headers: { Authorization: 'Bearer <ORQ_API_KEY>' },
});

const sdk = new NodeSDK({
  resource: new Resource({
    'service.name': 'my-node-service',
    'service.version': '1.0.0',
  }),
  traceExporter,
});

sdk.start();

// Example span
import { context, trace } from '@opentelemetry/api';
const tracer = trace.getTracer('example');

const span = tracer.startSpan('demo-span');
try {
  // your app logic here
} finally {
  span.end();
}

​

Knowledge API: New Parameters

• vector_search: Semantic similarity search
• keyword_search: Exact text matching
• hybrid_search: Combined approach (default)
• Separated Top-K Controls:
  • top_k: Controls initial document retrieval (1-100)
  • rerank_config.top_k: Controls final result count after reranking (1-100)
• New agentic_rag_config parameter to customize AI-driven search behavior per request

• Flexible Search Strategy: Choose optimal search method per query
• Fine-tuned Relevance: Retrieve broadly, return precisely with independent top-k settings
• Performance Optimization: Better control over retrieval vs. quality trade-offs

• Users can now choose specific search strategies instead of relying on knowledge base defaults
• Better control over the retrieval-rerank pipeline allows for more relevant results by retrieving more documents initially but returning fewer, higher-quality results

​

Support for OpenAI’s image edit and variation APIs

curl --location 'https://api.orq.ai/v2/proxy/images/edits' \
--header 'Authorization: Bearer ORQ_API_KEY' \
--form 'model="openai/gpt-image-1"' \
--form 'image=@"/path/to/file"' \
--form 'image=@"/path/to/file"' \
--form 'prompt="Add the style of card with the gradient style of the first image"'

import base64
from openai import OpenAI

client = OpenAI(base_url="https://api.orq.ai/v2/proxy/image/edit", api_key="ORQ_API_KEY")

prompt = """
Generate a photorealistic image of a gift basket on a white background
labeled 'Relax & Unwind' with a ribbon and handwriting-like font,
containing all the items in the reference pictures.
"""

result = client.images.edit(
    model="gpt-image-1",
    image=[
        open("body-lotion.png", "rb"),
        open("bath-bomb.png", "rb"),
        open("incense-kit.png", "rb"),
        open("soap.png", "rb"),
    ],
    prompt=prompt,
)

image_base64 = result.data[0].b64_json
image_bytes = base64.b64decode(image_base64)

# Save the image to a file
with open("gift-basket.png", "wb") as f:
    f.write(image_bytes)

import fs from "fs";
import OpenAI, { toFile } from "openai";

const client = new OpenAI({
  baseURL: "https://api.orq.ai/v2/proxy/image/edit",
  apiKey: process.env.ORQ_API_KEY
});

const imageFiles = [
    "bath-bomb.png",
    "body-lotion.png",
    "incense-kit.png",
    "soap.png",
];

const images = await Promise.all(
    imageFiles.map(async (file) =>
        await toFile(fs.createReadStream(file), null, {
            type: "image/png",
        })
    ),
);

const rsp = await client.images.edit({
    model: "gpt-image-1",
    image: images,
    prompt: "Create a lovely gift basket with these four items in it",
});

// Save the image to a file
const image_base64 = rsp.data[0].b64_json;
const image_bytes = Buffer.from(image_base64, "base64");
fs.writeFileSync("basket.png", image_bytes);

​

File Uploads for Anthropic Models

• Claude Opus 4.1 (claude-opus-4-1-20250805)
• Claude Opus 4 (claude-opus-4-20250514)
• Claude Sonnet 4 (claude-sonnet-4-20250514)
• Claude Sonnet 3.7 (claude-3-7-sonnet-20250219)
• Claude Haiku 3.5 (claude-3-5-haiku-20241022)

​

Comprehensive Budget Controls & Cost Management

• Configure budgets with daily, weekly, monthly, or yearly reset cycles
• Automatic period boundary calculation with UTC timezone standardization
• Period-aligned cache TTL for optimal performance
• New Budgets API to create budgets programatically

• Prevent individual users from exceeding allocated AI budgets
• Set department-level spending limits for enterprise teams
• Control costs across multiple AI models and services

​

GPT-5 available in the Model Garden

​

Preview Model Support

​

Three new RAGAS Evaluators for enhanced RAG Experiments

​

1. Context Recall

• Reference (ground truth)
• Retrieved text

• When you want to know if your retrieval step is actually surfacing all necessary information for the LLM to answer correctly.

​

2. Noise Sensitivity

• User message
• Reference (ground truth)
• Retrieved text (with added noise)

• When you want to see if your LLM can still provide accurate answers, even when extra, unrelated information appears in the context.

​

3. Context Entities Recall

• Reference (ground truth)
• Retrieved text

• When you want to be sure your retrieval is capturing and surfacing all the essential entities needed to answer user queries correctly.

​

Flexbile Prompt Configuration for Experiments

• Systematically validate regression, improvements, and guardrail compliance
• Compare outputs for cost, latency, and consistency
• Run more realistic and effective experiments based on actual user input

​

Chunking API

• Token chunker: Splits your text based on token count. Great for keeping chunks small enough for LLMs and for consistent embedding sizes.
• Sentence chunker: Breaks your text at sentence boundaries, so each chunk stays readable and sentences remain intact.
• Recursive chunker: Chunks text by working down a hierarchy (paragraphs, then sentences, then words) to maintain document structure.
• Semantic chunker: Groups together sentences that are topically related, so each chunk makes sense on its own.
• SDPM chunker: Uses advanced skip-gram patterns to find natural split points, especially helpful for technical or structured documents.
• Agentic chunker: Uses an LLM to determine the best split points, ideal for complex documents that need intelligent segmentation.

📘 Try the Chunking API out for yourself here: Chunking

​

Evaluators API

• Evaluators Hub API: Run any evaluator on demand via API, without needing to attach it to an experiment or deployment. Ideal for integrating evaluator runs into your own pipelines or automated tools.
• Evaluators API: Create, retrieve, update, and delete evaluators directly via API. This allows you to fully manage your evaluators programmatically, instead of only through the UI.

• Decoupled Evaluation: Run and manage evaluators without relying on experiments or deployments.
• Full API Control: Create, update, and remove evaluators as your evaluation needs evolve.
• Pipeline Integration: Integrate both evaluator management and execution into your own data pipelines or CI/CD workflows.
• Faster Iteration: Quickly test, refine, and iterate on evaluators using code.
• Automation: Scale evaluations and management as needed for batch operations or continuous quality monitoring.

curl --request POST \
     --url https://api.orq.ai/v2/evaluators/ragas_faithfulness \
     --header 'accept: application/json' \
     --header 'authorization: Bearer $ORQ_API_KEY' \
     --header 'content-type: application/json' \
     --data '
{
  "model": "openai/gpt-4.1",
  "query": "How large can my carry on bag be?",
  "output": "Your carry-on bag must not exceed 22 x 14 x 9 inches (56 x 36 x 23 cm), including handles and wheels. There is no weight limit for carry-on bags, but you must be able to lift it into the overhead bin without assistance.",
  "retrievals": [
    "{ \"question\": \"What are the size and weight limits for carry-on bags?\", \"answer\": \"Your carry-on bag must not exceed 22 x 14 x 9 inches (56 x 36 x 23 cm), including handles and wheels. There is no weight limit for carry-on bags, but you must be able to lift it into the overhead bin without assistance.\" }",
    "{ \"question\": \"Are there bag or seasonal bag limitations?\", \"answer\": \"Learn about checked bag limitations on our website.\" }",
    "{ \"question\": \"What are your liability limitations for lost bags?\", \"answer\": \"Learn about our liability limitations and see items excluded from liability on our website.\" }",
    "{ \"question\": \"What should I do about delayed or damaged bags?\", \"answer\": \"Find out what to do about delayed or damaged bags on our website.\" }"
  ]
}
'

​

Connect LiteLLM models to Orq

1. Go to the Integrations tab, click LiteLLM, and enter your base URL and API keys.
2. In Model Garden, click Add Model, select Import from LiteLLM, and choose your model(s).

• No migration pain: Instantly connect your existing LiteLLM models, no rework needed.
• Keep your infra, add new superpowers: Keep LiteLLM as your gateway while gaining everything Orq offers for GenAI ops.
• Best of both worlds: For teams already invested in LiteLLM, this means you get prompt management, deployments, evaluation, and observability—on top of your existing setup.

​

Self onboard Vertex AI models

​

Custom Human Review

• Categorical: Create button options with your own labels, such as good/bad or saved/deleted.
• Range: Set up a custom scoring slider (for example, a scale from 0 to 100).
• Open field: Allow users to enter free-form feedback, perfect for more detailed comments.

​

How to collect feedback

• Orq Studio: Custom human review templates can be used directly within the Orq.ai platform, where colleagues and domain experts review AI outputs and provide structured feedback.
• API: Alternatively, developers can leverage the Orq API to expose these custom feedback forms in their own applications or products, allowing end users to submit feedback seamlessly.

​

Why this matters

​

How to create custom Human Review

​

Evaluator playground

​

Attach files directly to the model

• Better File Understanding: Models can directly analyze the content and structure of your PDFs—including text, images, tables, and diagrams - improving extraction quality and structured outputs.
• Cleaner Prompts: File data is no longer dumped into your system or user message, resulting in faster, less error-prone requests.

curl --request POST \
     --url https://api.orq.ai/v2/deployments/invoke \
     --header 'accept: application/json' \
     --header 'authorization: Bearer <your_orq_key>' \
     --header 'content-type: application/json' \
     --data '
{
  "key": "key",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "prompt"
        },
        {
          "type": "file",
          "file": {
						"filename" : "<filename>",
            "file_data": "data:application/pdf;base64,<base64-encoded-data>"
          }
        }
      ]
    }
  ]
}
'

generation = client.deployments.invoke(
    key="deployment_key",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "prompt"
                },
                {
                    "type": "file",
                    "file": {
                        "file_data": "data:application/pdf;base64,<base64-encoded-data>",
                        "filename": "filename"
                    }
                }
            ]
        }
    ],
    metadata={
        "user_id": "123",
        "session_id": "456",
    }
)

const generation = await client.deployments.invoke({
  key: 'deployment_key',
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'text',
          text: 'prompt'
        },
        {
          type: 'file',
          file: {
            fileData: 'data:application/pdf;base64,<base64-encoded-data>',
            filename: 'filename.pdf'
          }
        }
      ]
    }
  ],
  metadata: {
    user_id: '123',
    session_id: '456'
  }
});

​

Evaluator runs without new generation

• Quality: Evaluate historical responses to ensure you’re assessing the true source of truth, not just new outputs.
• Flexibility: Apply evaluations to both** single responses and full conversation chains**, adapting to your specific review needs.

1. Prepare your dataset including LLM outputs in the messages column.
2. Set up an experiment and select one or more evaluators - do not select a prompt.
3. The evaluator will analyze the responses already present in the “messages” column of your dataset.

​

Reasoning field now included in the API response

{
    "id": "01JWQZ1JB6PSJ1F8KSWE5771GR",
    "created": "2025-06-02T09:19:59.308Z",
    "object": "vision",
    "model": "claude-sonnet-4-20250514",
    "provider": "anthropic",
    "is_final": true,
    "finalized": "2025-06-02T09:20:05.615Z",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "",
                "tool_calls": [
                    {
                        "type": "function",
                        "id": "toolu_01A57N7TgYAqcFmiMY9Q7TT4",
                        "index": 0,
                        "function": {
                            "name": "AA_tool",
                            "arguments": "{\"question\":\"How large can my carry on bag be?\",\"answer\":\"Your carry-on bag must not exceed 22 x 14 x 9 inches (56 x 36 x 23 cm), including handles and wheels. There is no weight limit for carry-on bags, but you must be able to lift it into the overhead bin without assistance.\"}"
                        }
                    }
                ],
                "reasoning": "The user is asking about carry-on bag size limits for American Airlines. Let me check the knowledge base for relevant information.\n\nLooking at the knowledge base, I can see there's a question that directly addresses this: \"What are the size and weight limits for carry-on bags?\" with the answer: \"Your carry-on bag must not exceed 22 x 14 x 9 inches (56 x 36 x 23 cm), including handles and wheels. There is no weight limit for carry-on bags, but you must be able to lift it into the overhead bin without assistance.\"\n\nThis question is clearly related to American Airlines travel and I have the relevant information in the knowledge base to answer it. I should use the AA_tool to structure my response.",
                "reasoning_signature": "Euxxx",
                "type": "tool_calls"
            },
            "finish_reason": "tool_use"
        }
    ],
    "provider_response": {
        "id": "msg_01VDbNW9N2khGgx16x5texpt",
        "type": "message",
        "role": "assistant",
        "model": "claude-sonnet-4-20250514",
        "content": [
            {
                "type": "thinking",
                "thinking": "The user is asking about carry-on bag size limits for American Airlines. Let me check the knowledge base for relevant information.\n\nLooking at the knowledge base, I can see there's a question that directly addresses this: \"What are the size and weight limits for carry-on bags?\" with the answer: \"Your carry-on bag must not exceed 22 x 14 x 9 inches (56 x 36 x 23 cm), including handles and wheels. There is no weight limit for carry-on bags, but you must be able to lift it into the overhead bin without assistance.\"\n\nThis question is clearly related to American Airlines travel and I have the relevant information in the knowledge base to answer it. I should use the AA_tool to structure my response.",
                "signature": "Euxxx"
            },
            {
                "type": "tool_use",
                "id": "toolu_01A57N7TgYAqcFmiMY9Q7TT4",
                "name": "AA_tool",
                "input": {
                    "question": "How large can my carry on bag be?",
                    "answer": "Your carry-on bag must not exceed 22 x 14 x 9 inches (56 x 36 x 23 cm), including handles and wheels. There is no weight limit for carry-on bags, but you must be able to lift it into the overhead bin without assistance."
                }
            }
        ],
        "stop_reason": "tool_use",
        "stop_sequence": null,
        "usage": {
            "input_tokens": 984,
            "cache_creation_input_tokens": 0,
            "cache_read_input_tokens": 0,
            "output_tokens": 312,
            "service_tier": "standard"
        }
    }
}

​

Experiment Runs overview

​

Improved Experiment configuration panel

​

Metadata filter on Knowledge Base Chunks

• Add custom metadata to each Knowledge Base chunk at creation.
• Search using metadata filters to refine results.

​

Why this matters

​

Example

curl --location 'http://my.orq.ai/v2/knowledge/{knowledge_id}/search' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $ORQ_API_KEY' \
--data '{
    "query": "What we the top editions of science fiction books",
    "filter": {
        "edition": {
            "gte": 2020
        }
    },
    "search_options":{
        "include_metadata": true,
        "include_vectors": true,
        "include_scores": true
    }
}'

from orq_ai_sdk import Orq
import os

client = Orq(api_key=os.getenv("ORQ_API_KEY"))

client.knowledge.search(
    knowledge_id="unique_knowledge_id",
    query={"edition": {"$gte": 2020}},
    search_options={
        "include_metadata": True,
        "include_vectors": True,
        "include_scores": True,
    },
)

import { Orq } from '@orq-ai/node';

const orq = new Orq({
  apiKey: 'ORQ_API_KEY',
});

orq.knowledge.search({
  knowledgeId: 'unique_knowledge_id',
  query: { edition: { $gte: 2020 } },
  searchOptions: {
    includeMetadata: true,
    includeVectors: true,
    includeScores: true,
  },
});

📘 To learn more about search, see Searching a knowledge base and the related API.

​

Claude 4 Sonnet & Opus

​

Tool response in Playground

​

OpenAI’s new o3, o4-mini, GPT-4.1 models

​

When to use what model

• o-series models are purpose-built for reasoning and tool-use. Choose o3 when raw capability beats everything else; use o4-mini when you still need strong reasoning but want 10× cheaper tokens and snappier replies.
• GPT-4.1 family is the evolution of the “generalist” GPT line. The full model gives you 1 M-token context with strong instruction following; mini trims cost/latency for mainstream workloads; nano pushes pricing and speed to the floor for simple or massive-scale tasks.

​

Rate Limits

​

Agentic RAG

• Document Grading Agent – determines if relevant chunks were retrieved.
• Query Refinement Agent – rewrites the query if needed.

​

Threads

• Start time and last update
• Total duration
• Number of traces (steps)
• Total cost and tokens used
• Project name and session ID
• Custom tags to help categorize threads

• Project
• Tags
• Start date

📘 To start using the Threads function add a thread id during the deployment invocation, to learn more, see Threads.

​

Jina AI - new embedding and rerank models

• Multilingual Embedding Models: Use Jina AI’s embedding models—like jina-embeddings-v3, supporting 89 languages and sequences up to 8,192 tokens.
• Efficient Rerank Models: Choose from three reranker variations:
  • Tiny – optimized for speed and low memory use.
  • Turbo – a balance between speed and accuracy.
  • Base – best-in-class accuracy for critical use cases.

​

Knowledge Base API

📘 Read more on the Knowledge API in our docs or in the SDKs

​

Hub

​

Tracing

• Retrieval – See which knowledge chunks were fetched.
• Embedding & Reranking – Understand how inputs are processed and prioritized.
• LLM Calls – Track prompts, responses, and latency.
• Evaluation & Guardrails – Ensure quality control in real time.
• Cache Usage – Spot inefficiencies in repeated queries.
• Fallbacks & Retries – Detect when your system auto-recovers from failures.

​

Billing impact - Event count

​

Python Evaluators

​

Benefits of Using Python for Evaluators

• Flexibility & Customization

• Seamless Integration

• Preloaded with NumPy

• Automated & Scalable

📘 Read here how to set up a custom Python Evaluator, see our docs

​

Move entities

​

Key Details:

• Move entities and directories freely between projects and directories.
• Maintain project-level permissions—entities will only be visible to the selected teams assigned to that project.
• Improve workflow efficiency by keeping related entities together where they belong.

​

How It Works

​

GPT 4.5

​

Why It Matters

​

What’s New?

​

Additional Enhancements

• Training Innovations: GPT-4.5 combines traditional supervised fine-tuning with scalable alignment, where smaller models generate high-quality training data for larger models. This approach accelerates training and improves the model’s ability to follow nuanced instructions.
• Multilingual and Multimodal Support: The model shows moderate gains in understanding multiple languages and processing various data types, making it versatile for global applications.

​

Performance & Analysis Tools

• Benchmark Evaluations: GPT-4.5 demonstrates solid improvements over GPT-4o, especially in math (+27.4%) and science (+17.8%), making it more reliable for factual reasoning.
• User Feedback Integration: OpenAI encourages users to engage with GPT-4.5 and provide feedback to further refine its capabilities and address any limitations.
• Complex reasoning: Even though GPT-4.5 is an improvement over GPT-4.o, for more complex reasoning tasks, you should use o3-mini.

​

Claude 3.7 Sonnet

🚧 We don’t return the thinking message yet, only the final output will be shown

📘 The model is available through Anthropic and Vertex AI

🚧 In AWS, the model is only available in the US region

​

Thinking - Budget tokens Parameter

​

Key highlights

• First-of-its-kind AI that can provide instant responses or extended, step-by-step reasoning based on user preference.
• API users can control “thinking time” to balance speed, accuracy, and cost.

• Improved ability to analyze charts, graphs, and complex diagrams, making it ideal for data science, analytics, and business intelligence.

• Enhanced reasoning and natural conversation flow make it ideal for interactive AI assistants that connect data, make decisions, and take actions.

• Automate complex workflows and repetitive tasks with industry-leading instruction following and tool integration.

• Supports a 200K context window for deep comprehension of long documents, knowledge bases, and large datasets.
• 128K output tokens (beta)—15x longer responses, enabling extensive code generation, planning, and detailed analysis.

​

Benchmarks and performance

​

Increased fallbacks

​

What’s New?

• Up to Five Fallback Models – Previously limited to one fallback model, users can now configure up to five fallback models for increased reliability.
• Flexible Fallback Configuration – The ability to define multiple fallback models allows for more advanced use cases, such as distributing API requests across different endpoints. For example:
  • Azure provides different regional deployments for models like GPT-4o. If a user is making a large number of API calls through the Sweden Central region, they can configure their first fallback to West Europe, and subsequent fallbacks to other locations if rate limits are hit.
  • If a primary model encounters rate limits, Orq can seamlessly transition requests to alternative endpoints, ensuring uninterrupted performance.

​

Importing prompts

​

What’s New?

• Seamless Importing – Users can now import their saved prompts directly into their workspace.
• Version Control – Select the specific version of a prompt to use.
• Model Configuration – You can import your prompt including the optional model config.

​

Datasets v2

​

What’s Changed?

​

Why This Change?

• Eliminates confusion between datasets and variable collections.
• Creates a unified structure for better dataset management.
• Ensures consistency for experiments and historical logging.

​

Why use Datasets v2?

• Upload a dataset to evaluate your use case in an Experiment
• Save human-curated logs into curated datasets, which you can use to fine-tune your model.
• Save bad-rated logs, crucial for prompt engineers looking to refine prompts and improve output quality.

​

Experiments v2 - Evaluate your LLM Config

​

Why Experiments Matter

• A/B testing – Compare different prompts or model configurations to determine what works best.
• Regression testing – Detect unintended changes in outputs after modifying prompts or configurations.
• Backtesting – Assess how new setups would have performed on historical data.

​

What’s New in Experiments V2?

• Compare Prompts Side by Side – Users can now directly compare multiple prompts within an experiment, making it easier to test variations and find the most effective approach.
• Intuitive and Flexible UI – The new interface is streamlined for ease of use, making configuration and analysis smoother than ever.
• Merged Data Sets and Variable Collections – Previously separate, these have now been combined to simplify workflows. We appreciate all the feedback on this and have made it clearer: when setting up an experiment, users now upload a data set containing input variables, messages (prompts), and expected outputs (if evaluators require a reference).

​

DeepSeek Models Now Available: 67B Chat, R1, and V3

​

What model should I use?

• Choose 67B Chat for fast everyday conversations
• Pick R1 when you need deeper thinking and logic for complex tasks
• Use V3 as a starting point to create your own specialized AI model

​

OpenAI’s Latest Small Reasoning Model – o3-mini

​

Llama 3.3 70b & Llama Guard 3 are now available through Together AI

​

Together AI

​

Llama Guard 8b

• How it works: It evaluates inputs and outputs for safety, returning a simple safe or unsafe verdict.
• Key feature: When flagged as unsafe, it identifies the violated content categories, offering clear insights.
• Why use it: Ideal for organizations exposing AI features to external users, it prevents misuse and ensures secure, compliant interactions.

​

Llama 3.3 70b

​

New Layout with Project Structure

​

What are the benefits of the new Project Structure UI?

• Easier grouping of resources that you’re working on as a team
• Less clutter when working on multiple use cases
• Better incorporation of teams and roles within a project

​

Where are my old resources?

​

Example structure

​

Can I move my entities to a different project or directory?

​

Search your entities

📘 To read more about our new project structure, please visit our documentation.

​

Online Guardrails in Live Deployments

• Guardrails: Block unacceptable inputs or outputs based on predefined criteria. If a guardrail is triggered, it can either retry the request or activate a fallback model, if configured.

​

What Are Guardrails?

• Threshold Configuration: Set customizable thresholds based on your specific use case to block inputs or outputs that fail to meet predefined criteria.
• Input and Output Guardrails: Apply rules to both incoming data and generated outputs, ensuring comprehensive quality control at every stage.
• Synchronous Operation: Guardrails run synchronously and do not introduce additional latency to your workflows.
• Fallback Mechanisms: When triggered, guardrails can stop processing or activate fallback options such as retries or alternative models to maintain continuity.

​

Why Are Evals and Guardrails Important?

1. First, they address the inherent risks of LLMs by monitoring and ensuring output quality and safety.
2. Second, they help teams overcome one of AI development’s biggest challenges: understanding how updates impact performance. Without clear insights, the development loop can break, turning iteration into guesswork rather than a structured engineering process. Evaluators help you understand whether an update is an improvement or a regression and quickly drill down into good / bad examples.

​

What Are Evaluators?

• Example: Use cosine similarity to measure how closely an LLM-generated summary matches the reference text.

• Example: Evaluate whether a chatbot’s response aligns with the intended tone and emotional context without comparing it to a specific reference.

• Example: Verify that a chatbot retrieving insurance policy details provides relevant and accurate information while excluding unrelated content.

• Example: Check if a response includes “return policy” or stays within a 200-character limit for customer service use cases.

• Example: Call a compliance-check API to validate that AI-generated responses meet industry-specific guidelines.

• Example: Confirm a JSON file for user registration includes fields like “username,” “age,” and “email” in the correct format.

​

What Are Guardrails?

• Robustness and Security. Guardrails are of huge importance in the aspect of security and robustness of an LLM-based application. This includes vulnerabilities such as prompt injections, jail-breaking, data leakage, handling illegible or obfuscated content.
• Information and Evidence. Guardrails can be put in place for fact checking, detecting hallucinations, making sure the response does not contain incorrect or misrepresented information, does not consider irrelevant information, the response is supported by evidence.
• Ethics and Safety. Making sure the response does not lead to harmful consequences either directly or indirectly. A response should not exhibit problematic social biases or share protected or sensitive information, take into account copyright and plagiarism law, use 3rd party content by appropriately citing it, etc.
• Tool-specific functionalities. Making sure the responses are on-topic, of proper extensiveness, written in the suitable tone, use specified terminology, etc.

​

HTTP and JSON Evaluators and Guardrails

​

HTTP Evaluators and Guardrails

• Purpose: Enable evaluation using external APIs to perform checks like compliance, custom logic, or business-specific validations.
• How It Works: You define the URL, headers, and payload for HTTP Requests, and the evaluator expects either a boolean or numerical response in the format:

{ "type": "boolean", "value": true }

{ "type": "number", "value": 1 }

• Use Case Example: An HTTP Evaluator can validate content against regulatory guidelines by routing a compliance check to your API, returning a boolean to accept or block the output. If you want to integrate an HTTP Eval into your set up, you can find more information in HTTP Evaluators.
• Guardrails: Configurable to accept or deny calls based on the returned value, providing dynamic control in experiments and deployments.

​

JSON Evaluators and Guardrails

• Purpose: Validate input or output payloads during Experiments or Deployments using a predefined JSON Schema.
• How It Works: Enter a JSON Schema specifying required fields and their expected types. For example:

{
  "name": "ReceiptDataExtraction",
  "strict": true,
  "schema": {
    "type": "object",
    "properties": {
      "Date": {
        "type": "string",
        "description": "Transaction date (YYYY-MM-DD)."
      },
      "Vendor": {
        "type": "string",
        "description": "Name of the vendor."
      },
      "Amount": {
        "type": "number",
        "description": "Total transaction amount."
      }
    },
    "required": ["Date", "Vendor", "Amount"],
    "additionalProperties": false
  }
}

• Use Case Example: A JSON Evaluator can ensure that generated content includes mandatory fields like “title” and “length,” blocking invalid payloads. For a more extensive explanation on how to set this up in your Deployment, you can go here.
• Guardrails: Toggle guardrails to block outputs that don’t adhere to the schema, ensuring compliance during Deployments or Experiments.

​

Master Your RAG with RAGAS Evals

• Out-of-the-Box Functionality: RAGAS Evaluators are ready to use and cannot be reconfigured, offering a consistent evaluation framework.
• Reference-Based Scoring: Some evaluators require a reference to calculate metrics like accuracy or faithfulness.
• Scoring Scale: Evaluations return a score between 0 and 1, with higher scores indicating better performance (e.g., higher relevance or faithfulness).

• Context Precision: Assesses how well retrieved chunks align with the user’s query.
  • Example: Ensures chunks about “home insurance” are prioritized when the query asks about coverage, filtering out irrelevant topics like auto insurance.
• Response Relevancy: Evaluates how directly the generated answer addresses the query.
  • Example: For “What are the fees for international transfers?” it ensures the answer is concise and focused on the fees without unrelated details.
• Faithfulness: Ensures the response is factually consistent with the retrieved context.
  • Example: For “What is the company’s remote work policy?” it checks if claims (e.g., “three days remote”) match the policy document.
• Context Entity Recall: Verifies that critical entities from the reference answer are included in the retrieved content.
  • Example: For “Tell me about the Taj Mahal,” ensures entities like “Shah Jahan” and “Agra” are retrieved.
• Context Recall: Measures if all necessary details from a reference are retrieved.
  • Example: For “What are the main benefits of product X?” it ensures all benefits like “cost savings” and “improved efficiency” are included.
• Noise Sensitivity: Checks if the system ignores irrelevant information in the retrieved context.
  • Example: For “What is LIC known for?” it ensures the response focuses on LIC’s attributes, filtering out unrelated economic data.
  

​

Evaluator Library: 50+ Ready-to-Use and Tailorable Evaluators

• The library includes 50+ preconfigured evaluation templates tailored for a wide range of use cases, from quality checks to compliance and contextual alignment.
• These templates are organized into categories such as LLM Evaluator Ragas Evaluator and Function Evaluator checks to help you find the right tool for your needs quickly.

• Each preconfigured evaluator can be customized and configured to fit specific project requirements.
  Once tailored, evaluators can be added to your personal Library for reuse in Experiment and Deployment.

• Evaluators are designed to work effortlessly in both experiments and deployments, enabling consistent and reliable quality control across development stages.

​

Improved LLMs as a Judge

• When you configure the output type as a number, boolean, or other types, the LLM ensures the result adheres strictly to the defined format.
• Additionally, each output includes a clear, concise explanation to support transparency and reliability in the evaluation process.

• The LLM-as-a-Judge now seamlessly integrates into live deployments, enabling asynchronous quality evaluation with a configurable sample rate. This approach provides actionable insights without disrupting workflows, ensuring consistent quality and minimizing drift.
• The LLM-as-a-Judge also supports Guardrail Mode, actively enforcing quality standards by blocking inputs or outputs that fail to meet defined criteria.

​

Online Evaluators in Live Deployments

​

Why Are Evaluators Important?

• Quality and Safety: Evaluators mitigate LLM risks by monitoring and ensuring output quality and safety.
• Performance Insights: They provide clear feedback on updates, helping teams identify improvements or regressions and refine iterations effectively.

​

What Are Evaluators?

• **Sample Rate Option:**Run evaluations on a subset of calls (e.g., 10%) to balance coverage and efficiency.
• **Cost Management:**Reduces additional expenses associated with running evaluators, especially LLM-based ones.
• Latency Control: Minimizes added response time by limiting the number of calls evaluated.

​

Cache your LLM response

1. Cost Savings

2. Time Savings

3. Improved Consistency

​

Structured Outputs on Azure

📘 Read more about Structured Outputs in Using Response Format.

​

Claude 3.5 Haiku and Sonnet (new)

​

Role Based Access Control improvements

• Admin: Complete access across all platform features, including user management, workspace settings, model configurations, and billing.
• Developer: Provides technical access to all essential features, including the ability to deploy changes within Deployments. Developers cannot manage models, billing, or users.
• Researcher: Offers access to Playgrounds, Experiments, and Deployments for research purposes. Researchers can review and annotate logs but cannot modify or deploy changes.

📘 Read more on how to set up permissions in our documentation

​

Attach files to a Deployment

📘 We support pdf, txt, docx, csv, xls - 10mb max

🚧 Be aware that we only extract the raw data, no images or objects. Also, encrypted files are not supported.

​

When should I attach a file and not use a Knowledge Base?

1. The need for full context understanding

2. Dynamic document context

3. Private or sensitive data

​

How to use it?

1. Upload your files

curl --location 'https://my.orq.ai/v2/files' \
--header 'Authorization: Bearer xxxxxx' \
--form 'purpose="retrieval"' \
--form 'file=@"/Users/cormick/Downloads/filename.pdf"'

2. Receive the _id

{
    "_id": "file_01JA5D27ZVW2N702Z0D3B1G8EK",
    "object_name": "files-api/workspaces/e747f6ac-19b0-47cd-8e79-0e1bf72b2a3e/retrieval/file_01JA5D27ZVW2N702Z0D3B1G8EK.vnd.openxmlformats-officedocument.spreadsheetml.sheet",
    "purpose": "retrieval",
    "file_name": "file_01JA5D27ZVW2N702Z0D3B1G8EK.vnd.openxmlformats-officedocument.spreadsheetml.sheet",
    "bytes": 5295,
    "created": "2024-10-14T11:36:54.189Z"
}

3. Send the file _id in addition to your normal Orq API call

curl --location 'https://my.orq.ai/v2/deployments/invoke' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer xxxxx' \
--data '{
    "key": "deployment_key",
    "messages": [
        {
            "role": "user",
            "content": ""
        }
    ],
    "file_ids": [
        "file_01JA5D27ZVW2N702Z0D3B1G8EK"
    ]
}'

​

Log Filtering

• Optimize Prompts: Filter logs with low ratings to pinpoint where your prompts need improvement and refine them for better outcomes.
• Troubleshoot Issues: Filter logs with error statuses to identify what went wrong and address any issues in your setup.
• Monitor Fallbacks: Filter logs that triggered a fallback model to see where the primary model didn’t perform as expected.

📘 Read more on our documentation page: Filters

​

Prompt Snippets

​

Using a Prompt Snippet

​

Configuring a Prompt Snippet

​

Version control

❗️ Please note that if you modify the snippet, all deployments containing that snippet will be automatically updated

📘 Read more on our documentation: Prompt.

​

Output Masking

​

Enabling Output Masking

📘 Read more on how to set up Output Masking in our Deployment Security and Privacy documentation

​

Workspace usage metrics

​

Structured outputs

​

Why should you use Structured Outputs?

• Reliable format: You don’t have to check or fix responses using the wrong format anymore.
• Clear refusals: When the model refuses something for safety reasons, it’s now easily detectable programmatically.
• Easier prompts: You don’t need to prompt the model to adhere to the schema anymore.

​

Using Structured Outputs

1. reponse_format (recommended) - In the dropdown menu (see GIF below), you can select the JSON pill and specify your schema.
2. Function Calling - In the tool, you can toggle the functionality of the structured output on/off.

• gpt-4o 2024-08-06 and later
• gpt-4o-mini 2024-07-18 and later

🚧 Reminder: To use Structured Outputs, all fields must be specified as required.

​

Vertex AI custom integration

​

Improved authentication

🚧 We removed the email and password login combination to enhance the security of our platform.

1. Magic Link Sign-In: Enter your email address, and we’ll send you a login link with a code for easy access. This means that you can also set up Two-Factor Authentication (2FA) or Multi-Factor Authentication (MFA) for added security.
2. Google Single Sign-On (SSO): Seamlessly log in using your Google account.
3. Microsoft Single Sign-On (SSO): Use your Microsoft account for quick and secure access.

​

New GoogleAI provider and Gemini models

📘 Google AI allows you to set up your own API keys, whereas VertexAI does not give you this option.

​

Why should I use Google AI instead of Vertex AI?

• Gemini-1.5-flash-8b-exp-0827
• Gemini-1.5-flash-exp-0827
• Gemini-1.5-pro-exp-0801
• Gemini-1.5-pro-exp-0827

​

Export Experiment Results

• Choose your file type: Select from CSV, JSON, or JSONL formats based on your specific requirements.
• Receive your data: After selecting your preferred format, your Experiment results will be compiled and sent to you via email, ready for further analysis.

• Leverage the exported data for custom visualizations, integrations, or further analysis in your preferred tools.
• Utilize the output of LLM generations in various formats for downstream tasks or additional finetuning.

​

Knowledge Bases & RAG

​

Why use a Knowledge Base?

​

Use cases

1. Reduced Hallucination: By relying on a well-structured knowledge base, the likelihood of the LLM generating incorrect or fabricated information (hallucinations) is significantly decreased.
2. Specific Context: A knowledge base allows for the inclusion of domain-specific or company-specific information, ensuring that the responses generated by the LLM are more aligned with the intended context.
3. Up-to-Date Information: Unlike static models, a knowledge base can be continuously updated with the latest information, providing the LLM with current and accurate data.

📘 Read more about Knowledge Bases in our Knowledge Base docs

​

Webhooks

• Automated data ingestion and processing: feeding real-time data from sources into Orq.
• Automated workflow orchestration: triggering specific deployments based on events.
• Logging: the ability to store the requests (input and output) back into your system.
• Monitoring and alerting: set up webhooks to alert relevant stakeholders about changes in model performance.

📘 Read more about webhooks in our docs.

​

Contact analytics

• Track Individual Users: Monitor costs and error rates for timely support.
• Track Internal Projects: Ensure projects stay within budget and meet goals.
• Track Client Projects: Get clear metrics per client projects.
• Track Specific Teams: Analyze team performance and optimize resource allocation.

📘 Read more on how to set up contact analytics in our docs

​

Onboard private models on Azure

​

Debug more effectively

​

Provider_response added in the SDK

{
  "id": "01J3MZHS6KFEWE7WBSAG0JP25X",
  "created": "2024-07-25T12:58:41.235Z",
  "object": "image",
  "model": "leonard-vision-xl",
  "provider": "leonardoai",
  "is_final": true,
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "url": "https://cdn.leonardo.ai/users/3bf1a64b-57f6-499b-907d-8b1286c1aa95/generations/dee000c8-7b6d-4d8c-82b4-d1992f8adce2/Default_a_tree_0.jpg"
      },
      "finish_reason": "stop"
    }
  ],
  "provider_response": {
    "generations_by_pk": {
      "generated_images": [
        {
          "url": "https://cdn.leonardo.ai/users/3bf1a64b-57f6-499b-907d-8b1286c1aa95/generations/dee000c8-7b6d-4d8c-82b4-d1992f8adce2/Default_a_tree_0.jpg",
          "nsfw": false,
          "id": "f47939ed-0841-48fa-b763-50e2e5b02e9a",
          "likeCount": 0,
          "motionMP4URL": null,
          "generated_image_variation_generics": []
        }
      ],
      "modelId": "5c232a9e-9061-4777-980a-ddc8e65647c6",
      "motion": null,
      "motionModel": null,
      "motionStrength": null,
      "prompt": "a tree",
      "negativePrompt": "",
      "imageHeight": 512,
      "imageToVideo": null,
      "imageWidth": 512,
      "inferenceSteps": 15,
      "seed": 4135326353,
      "public": false,
      "scheduler": "EULER_DISCRETE",
      "sdVersion": "SDXL_0_9",
      "status": "COMPLETE",
      "presetStyle": "CINEMATIC",
      "initStrength": null,
      "guidanceScale": null,
      "id": "dee000c8-7b6d-4d8c-82b4-d1992f8adce2",
      "createdAt": "2024-07-25T12:58:43.034",
      "promptMagic": false,
      "promptMagicVersion": null,
      "promptMagicStrength": null,
      "photoReal": false,
      "photoRealStrength": null,
      "fantasyAvatar": null,
      "generation_elements": []
    }
  }
}

​

GPT-4o mini

• Significantly smarter than GPT-3.5 Turbo, but not as capable as GPT-4o (see benchmark).
• 60% cheaper compared to GPT-3.5 Turbo.
• It’s a vision model, meaning it can interpret images.
• Context window is as big as GPT-4o (128k).

​

Claude 3.5 Sonnet

​

Prompt library

​

Llama 3 on Perplexity

• Generate up-to-date responses
• Retrieve dynamic data about the latest news etc.
• Understand real-world context better

​

Multi API key selector

​

Deployment test run

​

Groq custom integration

​

Keyboard shortcuts

​

New version pop-up

​

Tools workflow

​

New domain for ORQ.ai

🚧 If you’re using my.orquesta.dev please start using my.orq.ai