​

Introducing Evaluatorq

• Parallel Execution: Run multiple evaluation jobs concurrently with progress tracking
• Flexible Data Sources: Support for inline data, promises, and Orq platform datasets
• Type-safe: Fully written in TypeScript
• Orq Platform Integration: Seamlessly fetch and evaluate datasets from Orq AI (optional)

import { evaluatorq, job } from "@orq-ai/evaluatorq";

const textAnalyzer = job("text-analyzer", async (data) => {
  const text = data.inputs.text;
  const analysis = {
    length: text.length,
    wordCount: text.split(" ").length,
    uppercase: text.toUpperCase(),
  };

  return analysis;
});

await evaluatorq("text-analysis", {
  data: [
    { inputs: { text: "Hello world" } },
    { inputs: { text: "Testing evaluation" } },
  ],
  jobs: [textAnalyzer],
  evaluators: [
    {
      name: "length-check",
      scorer: async ({ output }) => {
        const passesCheck = output.length > 10;
        return {
          value: passesCheck ? 1 : 0,
          explanation: passesCheck
            ? "Output length is sufficient"
            : `Output too short (${output.length} chars, need >10)`,
        };
      },
    },
  ],
});

​

Native support for Gemini models - URL Context Tool

• Two-step retrieval: internal index cache first, live fetch if unavailable.
• Returns url_context_metadata with retrieval details and safety checks.
• Supports up to 20 URLs per request, max 34MB per URL.
• Retrieved content counts toward input tokens (tracked in usage_metadata).

• gemini-2.5-pro
• gemini-2.5-flash
• gemini-2.5-flash-lite

• Use direct, specific, fully qualified URLs (https://…).
• Avoid login-restricted or paywalled content.
• Combine with Google Search grounding for broader workflows.

​

OpenTelemetry Tracing Support

• Native OpenTelemetry (OTel) export for traces.
• Send spans directly to our OTLP/HTTP collector at: https://api.orq.ai/v2/otel.
• Works with standard OTel SDKs and exporters (Python, Node.js).
• Supports custom headers (e.g., bearer tokens) and resource attributes (e.g., service.name).

# Common env vars (both Python & Node.js)
export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.orq.ai/v2/otel"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <ORQ_API_KEY>"
export OTEL_RESOURCE_ATTRIBUTES="service.name=my-service,service.version=1.0.0"

# pip install opentelemetry-sdk opentelemetry-exporter-otlp-proto-http opentelemetry-instrumentation

from opentelemetry import trace
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

resource = Resource.create({
    "service.name": "my-python-service",
    "service.version": "1.0.0"
})

provider = TracerProvider(resource=resource)
trace.set_tracer_provider(provider)

exporter = OTLPSpanExporter(
    endpoint="https://api.orq.ai/v2/otel",
    # Optional if your setup requires auth
    headers={"Authorization": "Bearer <ORQ_API_KEY>"}
)

provider.add_span_processor(BatchSpanProcessor(exporter))

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("demo-span"):
    # your app logic here
    pass

import { NodeSDK } from '@opentelemetry/sdk-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { Resource } from '@opentelemetry/resources';

const traceExporter = new OTLPTraceExporter({
  url: 'https://api.orq.ai/v2/otel',
  // Optional if your setup requires auth
  headers: { Authorization: 'Bearer <ORQ_API_KEY>' },
});

const sdk = new NodeSDK({
  resource: new Resource({
    'service.name': 'my-node-service',
    'service.version': '1.0.0',
  }),
  traceExporter,
});

sdk.start();

// Example span
import { context, trace } from '@opentelemetry/api';
const tracer = trace.getTracer('example');

const span = tracer.startSpan('demo-span');
try {
  // your app logic here
} finally {
  span.end();
}

​

Knowledge API: New Parameters

• vector_search: Semantic similarity search
• keyword_search: Exact text matching
• hybrid_search: Combined approach (default)
• Separated Top-K Controls:
  • top_k: Controls initial document retrieval (1-100)
  • rerank_config.top_k: Controls final result count after reranking (1-100)
• New agentic_rag_config parameter to customize AI-driven search behavior per request

• Flexible Search Strategy: Choose optimal search method per query
• Fine-tuned Relevance: Retrieve broadly, return precisely with independent top-k settings
• Performance Optimization: Better control over retrieval vs. quality trade-offs

• Users can now choose specific search strategies instead of relying on knowledge base defaults
• Better control over the retrieval-rerank pipeline allows for more relevant results by retrieving more documents initially but returning fewer, higher-quality results

​

Support for OpenAI’s image edit and variation APIs

curl --location 'https://api.orq.ai/v2/proxy/images/edits' \
--header 'Authorization: Bearer ORQ_API_KEY' \
--form 'model="openai/gpt-image-1"' \
--form 'image=@"/path/to/file"' \
--form 'image=@"/path/to/file"' \
--form 'prompt="Add the style of card with the gradient style of the first image"'

import base64
from openai import OpenAI

client = OpenAI(base_url="https://api.orq.ai/v2/proxy/image/edit", api_key="ORQ_API_KEY")

prompt = """
Generate a photorealistic image of a gift basket on a white background
labeled 'Relax & Unwind' with a ribbon and handwriting-like font,
containing all the items in the reference pictures.
"""

result = client.images.edit(
    model="gpt-image-1",
    image=[
        open("body-lotion.png", "rb"),
        open("bath-bomb.png", "rb"),
        open("incense-kit.png", "rb"),
        open("soap.png", "rb"),
    ],
    prompt=prompt,
)

image_base64 = result.data[0].b64_json
image_bytes = base64.b64decode(image_base64)

# Save the image to a file
with open("gift-basket.png", "wb") as f:
    f.write(image_bytes)

import fs from "fs";
import OpenAI, { toFile } from "openai";

const client = new OpenAI({
  baseURL: "https://api.orq.ai/v2/proxy/image/edit",
  apiKey: process.env.ORQ_API_KEY
});

const imageFiles = [
    "bath-bomb.png",
    "body-lotion.png",
    "incense-kit.png",
    "soap.png",
];

const images = await Promise.all(
    imageFiles.map(async (file) =>
        await toFile(fs.createReadStream(file), null, {
            type: "image/png",
        })
    ),
);

const rsp = await client.images.edit({
    model: "gpt-image-1",
    image: images,
    prompt: "Create a lovely gift basket with these four items in it",
});

// Save the image to a file
const image_base64 = rsp.data[0].b64_json;
const image_bytes = Buffer.from(image_base64, "base64");
fs.writeFileSync("basket.png", image_bytes);

​

File Uploads for Anthropic Models

• Claude Opus 4.1 (claude-opus-4-1-20250805)
• Claude Opus 4 (claude-opus-4-20250514)
• Claude Sonnet 4 (claude-sonnet-4-20250514)
• Claude Sonnet 3.7 (claude-3-7-sonnet-20250219)
• Claude Haiku 3.5 (claude-3-5-haiku-20241022)

​

Comprehensive Budget Controls & Cost Management

• Configure budgets with daily, weekly, monthly, or yearly reset cycles
• Automatic period boundary calculation with UTC timezone standardization
• Period-aligned cache TTL for optimal performance
• New Budgets API to create budgets programatically

• Prevent individual users from exceeding allocated AI budgets
• Set department-level spending limits for enterprise teams
• Control costs across multiple AI models and services

​

GPT-5 available in the Model Garden

​

Preview Model Support

​

Three new RAGAS Evaluators for enhanced RAG Experiments

​

1. Context Recall

• Reference (ground truth)
• Retrieved text

• When you want to know if your retrieval step is actually surfacing all necessary information for the LLM to answer correctly.

​

2. Noise Sensitivity

• User message
• Reference (ground truth)
• Retrieved text (with added noise)

• When you want to see if your LLM can still provide accurate answers, even when extra, unrelated information appears in the context.

​

3. Context Entities Recall

• Reference (ground truth)
• Retrieved text

• When you want to be sure your retrieval is capturing and surfacing all the essential entities needed to answer user queries correctly.

​

Flexbile Prompt Configuration for Experiments

• Systematically validate regression, improvements, and guardrail compliance
• Compare outputs for cost, latency, and consistency
• Run more realistic and effective experiments based on actual user input

​

Chunking API

• Token chunker: Splits your text based on token count. Great for keeping chunks small enough for LLMs and for consistent embedding sizes.
• Sentence chunker: Breaks your text at sentence boundaries, so each chunk stays readable and sentences remain intact.
• Recursive chunker: Chunks text by working down a hierarchy (paragraphs, then sentences, then words) to maintain document structure.
• Semantic chunker: Groups together sentences that are topically related, so each chunk makes sense on its own.
• SDPM chunker: Uses advanced skip-gram patterns to find natural split points, especially helpful for technical or structured documents.
• Agentic chunker: Uses an LLM to determine the best split points, ideal for complex documents that need intelligent segmentation.

📘 Try the Chunking API out for yourself here: Chunking

​

Evaluators API

• Evaluators Hub API: Run any evaluator on demand via API, without needing to attach it to an experiment or deployment. Ideal for integrating evaluator runs into your own pipelines or automated tools.
• Evaluators API: Create, retrieve, update, and delete evaluators directly via API. This allows you to fully manage your evaluators programmatically, instead of only through the UI.

• Decoupled Evaluation: Run and manage evaluators without relying on experiments or deployments.
• Full API Control: Create, update, and remove evaluators as your evaluation needs evolve.
• Pipeline Integration: Integrate both evaluator management and execution into your own data pipelines or CI/CD workflows.
• Faster Iteration: Quickly test, refine, and iterate on evaluators using code.
• Automation: Scale evaluations and management as needed for batch operations or continuous quality monitoring.

curl --request POST \
     --url https://api.orq.ai/v2/evaluators/ragas_faithfulness \
     --header 'accept: application/json' \
     --header 'authorization: Bearer $ORQ_API_KEY' \
     --header 'content-type: application/json' \
     --data '
{
  "model": "openai/gpt-4.1",
  "query": "How large can my carry on bag be?",
  "output": "Your carry-on bag must not exceed 22 x 14 x 9 inches (56 x 36 x 23 cm), including handles and wheels. There is no weight limit for carry-on bags, but you must be able to lift it into the overhead bin without assistance.",
  "retrievals": [
    "{ \"question\": \"What are the size and weight limits for carry-on bags?\", \"answer\": \"Your carry-on bag must not exceed 22 x 14 x 9 inches (56 x 36 x 23 cm), including handles and wheels. There is no weight limit for carry-on bags, but you must be able to lift it into the overhead bin without assistance.\" }",
    "{ \"question\": \"Are there bag or seasonal bag limitations?\", \"answer\": \"Learn about checked bag limitations on our website.\" }",
    "{ \"question\": \"What are your liability limitations for lost bags?\", \"answer\": \"Learn about our liability limitations and see items excluded from liability on our website.\" }",
    "{ \"question\": \"What should I do about delayed or damaged bags?\", \"answer\": \"Find out what to do about delayed or damaged bags on our website.\" }"
  ]
}
'

​

Connect LiteLLM models to Orq

1. Go to the Integrations tab, click LiteLLM, and enter your base URL and API keys.
2. In Model Garden, click Add Model, select Import from LiteLLM, and choose your model(s).

• No migration pain: Instantly connect your existing LiteLLM models, no rework needed.
• Keep your infra, add new superpowers: Keep LiteLLM as your gateway while gaining everything Orq offers for GenAI ops.
• Best of both worlds: For teams already invested in LiteLLM, this means you get prompt management, deployments, evaluation, and observability—on top of your existing setup.

​

Self onboard Vertex AI models

​

Custom Human Review

• Categorical: Create button options with your own labels, such as good/bad or saved/deleted.
• Range: Set up a custom scoring slider (for example, a scale from 0 to 100).
• Open field: Allow users to enter free-form feedback, perfect for more detailed comments.

​

How to collect feedback

• Orq Studio: Custom human review templates can be used directly within the Orq.ai platform, where colleagues and domain experts review AI outputs and provide structured feedback.
• API: Alternatively, developers can leverage the Orq API to expose these custom feedback forms in their own applications or products, allowing end users to submit feedback seamlessly.

​

Why this matters

​

How to create custom Human Review

​

Evaluator playground

​

Attach files directly to the model

• Better File Understanding: Models can directly analyze the content and structure of your PDFs—including text, images, tables, and diagrams - improving extraction quality and structured outputs.
• Cleaner Prompts: File data is no longer dumped into your system or user message, resulting in faster, less error-prone requests.

curl --request POST \
     --url https://api.orq.ai/v2/deployments/invoke \
     --header 'accept: application/json' \
     --header 'authorization: Bearer <your_orq_key>' \
     --header 'content-type: application/json' \
     --data '
{
  "key": "key",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "prompt"
        },
        {
          "type": "file",
          "file": {
						"filename" : "<filename>",
            "file_data": "data:application/pdf;base64,<base64-encoded-data>"
          }
        }
      ]
    }
  ]
}
'

generation = client.deployments.invoke(
    key="deployment_key",
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "text",
                    "text": "prompt"
                },
                {
                    "type": "file",
                    "file": {
                        "file_data": "data:application/pdf;base64,<base64-encoded-data>",
                        "filename": "filename"
                    }
                }
            ]
        }
    ],
    metadata={
        "user_id": "123",
        "session_id": "456",
    }
)

const generation = await client.deployments.invoke({
  key: 'deployment_key',
  messages: [
    {
      role: 'user',
      content: [
        {
          type: 'text',
          text: 'prompt'
        },
        {
          type: 'file',
          file: {
            fileData: 'data:application/pdf;base64,<base64-encoded-data>',
            filename: 'filename.pdf'
          }
        }
      ]
    }
  ],
  metadata: {
    user_id: '123',
    session_id: '456'
  }
});

​

Evaluator runs without new generation

• Quality: Evaluate historical responses to ensure you’re assessing the true source of truth, not just new outputs.
• Flexibility: Apply evaluations to both** single responses and full conversation chains**, adapting to your specific review needs.

1. Prepare your dataset including LLM outputs in the messages column.
2. Set up an experiment and select one or more evaluators - do not select a prompt.
3. The evaluator will analyze the responses already present in the “messages” column of your dataset.

​

Reasoning field now included in the API response

{
    "id": "01JWQZ1JB6PSJ1F8KSWE5771GR",
    "created": "2025-06-02T09:19:59.308Z",
    "object": "vision",
    "model": "claude-sonnet-4-20250514",
    "provider": "anthropic",
    "is_final": true,
    "finalized": "2025-06-02T09:20:05.615Z",
    "choices": [
        {
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "",
                "tool_calls": [
                    {
                        "type": "function",
                        "id": "toolu_01A57N7TgYAqcFmiMY9Q7TT4",
                        "index": 0,
                        "function": {
                            "name": "AA_tool",
                            "arguments": "{\"question\":\"How large can my carry on bag be?\",\"answer\":\"Your carry-on bag must not exceed 22 x 14 x 9 inches (56 x 36 x 23 cm), including handles and wheels. There is no weight limit for carry-on bags, but you must be able to lift it into the overhead bin without assistance.\"}"
                        }
                    }
                ],
                "reasoning": "The user is asking about carry-on bag size limits for American Airlines. Let me check the knowledge base for relevant information.\n\nLooking at the knowledge base, I can see there's a question that directly addresses this: \"What are the size and weight limits for carry-on bags?\" with the answer: \"Your carry-on bag must not exceed 22 x 14 x 9 inches (56 x 36 x 23 cm), including handles and wheels. There is no weight limit for carry-on bags, but you must be able to lift it into the overhead bin without assistance.\"\n\nThis question is clearly related to American Airlines travel and I have the relevant information in the knowledge base to answer it. I should use the AA_tool to structure my response.",
                "reasoning_signature": "Euxxx",
                "type": "tool_calls"
            },
            "finish_reason": "tool_use"
        }
    ],
    "provider_response": {
        "id": "msg_01VDbNW9N2khGgx16x5texpt",
        "type": "message",
        "role": "assistant",
        "model": "claude-sonnet-4-20250514",
        "content": [
            {
                "type": "thinking",
                "thinking": "The user is asking about carry-on bag size limits for American Airlines. Let me check the knowledge base for relevant information.\n\nLooking at the knowledge base, I can see there's a question that directly addresses this: \"What are the size and weight limits for carry-on bags?\" with the answer: \"Your carry-on bag must not exceed 22 x 14 x 9 inches (56 x 36 x 23 cm), including handles and wheels. There is no weight limit for carry-on bags, but you must be able to lift it into the overhead bin without assistance.\"\n\nThis question is clearly related to American Airlines travel and I have the relevant information in the knowledge base to answer it. I should use the AA_tool to structure my response.",
                "signature": "Euxxx"
            },
            {
                "type": "tool_use",
                "id": "toolu_01A57N7TgYAqcFmiMY9Q7TT4",
                "name": "AA_tool",
                "input": {
                    "question": "How large can my carry on bag be?",
                    "answer": "Your carry-on bag must not exceed 22 x 14 x 9 inches (56 x 36 x 23 cm), including handles and wheels. There is no weight limit for carry-on bags, but you must be able to lift it into the overhead bin without assistance."
                }
            }
        ],
        "stop_reason": "tool_use",
        "stop_sequence": null,
        "usage": {
            "input_tokens": 984,
            "cache_creation_input_tokens": 0,
            "cache_read_input_tokens": 0,
            "output_tokens": 312,
            "service_tier": "standard"
        }
    }
}

​

Experiment Runs overview

​

Improved Experiment configuration panel

​

Metadata filter on Knowledge Base Chunks

• Add custom metadata to each Knowledge Base chunk at creation.
• Search using metadata filters to refine results.

​

Why this matters

​

Example

curl --location 'http://my.orq.ai/v2/knowledge/{knowledge_id}/search' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer $ORQ_API_KEY' \
--data '{
    "query": "What we the top editions of science fiction books",
    "filter": {
        "edition": {
            "gte": 2020
        }
    },
    "search_options":{
        "include_metadata": true,
        "include_vectors": true,
        "include_scores": true
    }
}'

from orq_ai_sdk import Orq
import os

client = Orq(api_key=os.getenv("ORQ_API_KEY"))

client.knowledge.search(
    knowledge_id="unique_knowledge_id",
    query={"edition": {"$gte": 2020}},
    search_options={
        "include_metadata": True,
        "include_vectors": True,
        "include_scores": True,
    },
)

import { Orq } from '@orq-ai/node';

const orq = new Orq({
  apiKey: 'ORQ_API_KEY',
});

orq.knowledge.search({
  knowledgeId: 'unique_knowledge_id',
  query: { edition: { $gte: 2020 } },
  searchOptions: {
    includeMetadata: true,
    includeVectors: true,
    includeScores: true,
  },
});

📘 To learn more about search, see Searching a knowledge base and the related API.

​

Claude 4 Sonnet & Opus

​

Tool response in Playground

​

OpenAI’s new o3, o4-mini, GPT-4.1 models

​

When to use what model

• o-series models are purpose-built for reasoning and tool-use. Choose o3 when raw capability beats everything else; use o4-mini when you still need strong reasoning but want 10× cheaper tokens and snappier replies.
• GPT-4.1 family is the evolution of the “generalist” GPT line. The full model gives you 1 M-token context with strong instruction following; mini trims cost/latency for mainstream workloads; nano pushes pricing and speed to the floor for simple or massive-scale tasks.

​

Rate Limits

​

Agentic RAG

• Document Grading Agent – determines if relevant chunks were retrieved.
• Query Refinement Agent – rewrites the query if needed.

​

Threads

• Start time and last update
• Total duration
• Number of traces (steps)
• Total cost and tokens used
• Project name and session ID
• Custom tags to help categorize threads

• Project
• Tags
• Start date

📘 To start using the Threads function add a thread id during the deployment invocation, to learn more, see Threads.

​

Jina AI - new embedding and rerank models

• Multilingual Embedding Models: Use Jina AI’s embedding models—like jina-embeddings-v3, supporting 89 languages and sequences up to 8,192 tokens.
• Efficient Rerank Models: Choose from three reranker variations:
  • Tiny – optimized for speed and low memory use.
  • Turbo – a balance between speed and accuracy.
  • Base – best-in-class accuracy for critical use cases.

​

Knowledge Base API

📘 Read more on the Knowledge API in our docs or in the SDKs

​

Hub

​

Tracing

• Retrieval – See which knowledge chunks were fetched.
• Embedding & Reranking – Understand how inputs are processed and prioritized.
• LLM Calls – Track prompts, responses, and latency.
• Evaluation & Guardrails – Ensure quality control in real time.
• Cache Usage – Spot inefficiencies in repeated queries.
• Fallbacks & Retries – Detect when your system auto-recovers from failures.

​

Billing impact - Event count

​

Python Evaluators

​

Benefits of Using Python for Evaluators

• Flexibility & Customization

• Seamless Integration

• Preloaded with NumPy

• Automated & Scalable

📘 Read here how to set up a custom Python Evaluator, see our docs

​

Move entities

​

Key Details:

• Move entities and directories freely between projects and directories.
• Maintain project-level permissions—entities will only be visible to the selected teams assigned to that project.
• Improve workflow efficiency by keeping related entities together where they belong.

​

How It Works

​

GPT 4.5

​

Why It Matters

​

What’s New?

​

Additional Enhancements

• Training Innovations: GPT-4.5 combines traditional supervised fine-tuning with scalable alignment, where smaller models generate high-quality training data for larger models. This approach accelerates training and improves the model’s ability to follow nuanced instructions.
• Multilingual and Multimodal Support: The model shows moderate gains in understanding multiple languages and processing various data types, making it versatile for global applications.

​

Performance & Analysis Tools

• Benchmark Evaluations: GPT-4.5 demonstrates solid improvements over GPT-4o, especially in math (+27.4%) and science (+17.8%), making it more reliable for factual reasoning.
• User Feedback Integration: OpenAI encourages users to engage with GPT-4.5 and provide feedback to further refine its capabilities and address any limitations.
• Complex reasoning: Even though GPT-4.5 is an improvement over GPT-4.o, for more complex reasoning tasks, you should use o3-mini.

​

Claude 3.7 Sonnet

🚧 We don’t return the thinking message yet, only the final output will be shown

📘 The model is available through Anthropic and Vertex AI

🚧 In AWS, the model is only available in the US region

​

Thinking - Budget tokens Parameter

​

Key highlights

• First-of-its-kind AI that can provide instant responses or extended, step-by-step reasoning based on user preference.
• API users can control “thinking time” to balance speed, accuracy, and cost.

• Improved ability to analyze charts, graphs, and complex diagrams, making it ideal for data science, analytics, and business intelligence.

• Enhanced reasoning and natural conversation flow make it ideal for interactive AI assistants that connect data, make decisions, and take actions.

• Automate complex workflows and repetitive tasks with industry-leading instruction following and tool integration.

• Supports a 200K context window for deep comprehension of long documents, knowledge bases, and large datasets.
• 128K output tokens (beta)—15x longer responses, enabling extensive code generation, planning, and detailed analysis.

​

Benchmarks and performance

​

Increased fallbacks

​

What’s New?

• Up to Five Fallback Models – Previously limited to one fallback model, users can now configure up to five fallback models for increased reliability.
• Flexible Fallback Configuration – The ability to define multiple fallback models allows for more advanced use cases, such as distributing API requests across different endpoints. For example:
  • Azure provides different regional deployments for models like GPT-4o. If a user is making a large number of API calls through the Sweden Central region, they can configure their first fallback to West Europe, and subsequent fallbacks to other locations if rate limits are hit.
  • If a primary model encounters rate limits, Orq can seamlessly transition requests to alternative endpoints, ensuring uninterrupted performance.

​

Importing prompts

​

What’s New?

• Seamless Importing – Users can now import their saved prompts directly into their workspace.
• Version Control – Select the specific version of a prompt to use.
• Model Configuration – You can import your prompt including the optional model config.

​

Datasets v2

​

What’s Changed?

​

Why This Change?

• Eliminates confusion between datasets and variable collections.
• Creates a unified structure for better dataset management.
• Ensures consistency for experiments and historical logging.

​

Why use Datasets v2?

• Upload a dataset to evaluate your use case in an Experiment
• Save human-curated logs into curated datasets, which you can use to fine-tune your model.
• Save bad-rated logs, crucial for prompt engineers looking to refine prompts and improve output quality.

​

Experiments v2 - Evaluate your LLM Config

​

Why Experiments Matter

• A/B testing – Compare different prompts or model configurations to determine what works best.
• Regression testing – Detect unintended changes in outputs after modifying prompts or configurations.
• Backtesting – Assess how new setups would have performed on historical data.

​

What’s New in Experiments V2?

• Compare Prompts Side by Side – Users can now directly compare multiple prompts within an experiment, making it easier to test variations and find the most effective approach.
• Intuitive and Flexible UI – The new interface is streamlined for ease of use, making configuration and analysis smoother than ever.
• Merged Data Sets and Variable Collections – Previously separate, these have now been combined to simplify workflows. We appreciate all the feedback on this and have made it clearer: when setting up an experiment, users now upload a data set containing input variables, messages (prompts), and expected outputs (if evaluators require a reference).

​

DeepSeek Models Now Available: 67B Chat, R1, and V3

​

What model should I use?

• Choose 67B Chat for fast everyday conversations
• Pick R1 when you need deeper thinking and logic for complex tasks
• Use V3 as a starting point to create your own specialized AI model

​

OpenAI’s Latest Small Reasoning Model – o3-mini

​

Llama 3.3 70b & Llama Guard 3 are now available through Together AI

​

Together AI

​

Llama Guard 8b

• How it works: It evaluates inputs and outputs for safety, returning a simple safe or unsafe verdict.
• Key feature: When flagged as unsafe, it identifies the violated content categories, offering clear insights.
• Why use it: Ideal for organizations exposing AI features to external users, it prevents misuse and ensures secure, compliant interactions.

​

Llama 3.3 70b

​

New Layout with Project Structure

​

What are the benefits of the new Project Structure UI?

• Easier grouping of resources that you’re working on as a team
• Less clutter when working on multiple use cases
• Better incorporation of teams and roles within a project

​

Where are my old resources?

​

Example structure

​

Can I move my entities to a different project or directory?

​

Search your entities

📘 To read more about our new project structure, please visit our documentation.

​

Online Guardrails in Live Deployments

• Guardrails: Block unacceptable inputs or outputs based on predefined criteria. If a guardrail is triggered, it can either retry the request or activate a fallback model, if configured.

​

What Are Guardrails?

• Threshold Configuration: Set customizable thresholds based on your specific use case to block inputs or outputs that fail to meet predefined criteria.
• Input and Output Guardrails: Apply rules to both incoming data and generated outputs, ensuring comprehensive quality control at every stage.
• Synchronous Operation: Guardrails run synchronously and do not introduce additional latency to your workflows.
• Fallback Mechanisms: When triggered, guardrails can stop processing or activate fallback options such as retries or alternative models to maintain continuity.

​

Why Are Evals and Guardrails Important?

1. First, they address the inherent risks of LLMs by monitoring and ensuring output quality and safety.
2. Second, they help teams overcome one of AI development’s biggest challenges: understanding how updates impact performance. Without clear insights, the development loop can break, turning iteration into guesswork rather than a structured engineering process. Evaluators help you understand whether an update is an improvement or a regression and quickly drill down into good / bad examples.

​

What Are Evaluators?

• Example: Use cosine similarity to measure how closely an LLM-generated summary matches the reference text.

• Example: Evaluate whether a chatbot’s response aligns with the intended tone and emotional context without comparing it to a specific reference.

• Example: Verify that a chatbot retrieving insurance policy details provides relevant and accurate information while excluding unrelated content.

• Example: Check if a response includes “return policy” or stays within a 200-character limit for customer service use cases.

• Example: Call a compliance-check API to validate that AI-generated responses meet industry-specific guidelines.

• Example: Confirm a JSON file for user registration includes fields like “username,” “age,” and “email” in the correct format.

​

What Are Guardrails?

• Robustness and Security. Guardrails are of huge importance in the aspect of security and robustness of an LLM-based application. This includes vulnerabilities such as prompt injections, jail-breaking, data leakage, handling illegible or obfuscated content.
• Information and Evidence. Guardrails can be put in place for fact checking, detecting hallucinations, making sure the response does not contain incorrect or misrepresented information, does not consider irrelevant information, the response is supported by evidence.
• Ethics and Safety. Making sure the response does not lead to harmful consequences either directly or indirectly. A response should not exhibit problematic social biases or share protected or sensitive information, take into account copyright and plagiarism law, use 3rd party content by appropriately citing it, etc.
• Tool-specific functionalities. Making sure the responses are on-topic, of proper extensiveness, written in the suitable tone, use specified terminology, etc.

​

HTTP and JSON Evaluators and Guardrails

​

HTTP Evaluators and Guardrails

• Purpose: Enable evaluation using external APIs to perform checks like compliance, custom logic, or business-specific validations.
• How It Works: You define the URL, headers, and payload for HTTP Requests, and the evaluator expects either a boolean or numerical response in the format:

{ "type": "boolean", "value": true }

{ "type": "number", "value": 1 }

• Use Case Example: An HTTP Evaluator can validate content against regulatory guidelines by routing a compliance check to your API, returning a boolean to accept or block the output. If you want to integrate an HTTP Eval into your set up, you can find more information in HTTP Evaluators.
• Guardrails: Configurable to accept or deny calls based on the returned value, providing dynamic control in experiments and deployments.

​

JSON Evaluators and Guardrails

• Purpose: Validate input or output payloads during Experiments or Deployments using a predefined JSON Schema.
• How It Works: Enter a JSON Schema specifying required fields and their expected types. For example:

{
  "name": "ReceiptDataExtraction",
  "strict": true,
  "schema": {
    "type": "object",
    "properties": {
      "Date": {
        "type": "string",
        "description": "Transaction date (YYYY-MM-DD)."
      },
      "Vendor": {
        "type": "string",
        "description": "Name of the vendor."
      },
      "Amount": {
        "type": "number",
        "description": "Total transaction amount."
      }
    },
    "required": ["Date", "Vendor", "Amount"],
    "additionalProperties": false
  }
}

• Use Case Example: A JSON Evaluator can ensure that generated content includes mandatory fields like “title” and “length,” blocking invalid payloads. For a more extensive explanation on how to set this up in your Deployment, you can go here.
• Guardrails: Toggle guardrails to block outputs that don’t adhere to the schema, ensuring compliance during Deployments or Experiments.

​

Master Your RAG with RAGAS Evals

• Out-of-the-Box Functionality: RAGAS Evaluators are ready to use and cannot be reconfigured, offering a consistent evaluation framework.
• Reference-Based Scoring: Some evaluators require a reference to calculate metrics like accuracy or faithfulness.
• Scoring Scale: Evaluations return a score between 0 and 1, with higher scores indicating better performance (e.g., higher relevance or faithfulness).

• Context Precision: Assesses how well retrieved chunks align with the user’s query.
  • Example: Ensures chunks about “home insurance” are prioritized when the query asks about coverage, filtering out irrelevant topics like auto insurance.
• Response Relevancy: Evaluates how directly the generated answer addresses the query.
  • Example: For “What are the fees for international transfers?” it ensures the answer is concise and focused on the fees without unrelated details.
• Faithfulness: Ensures the response is factually consistent with the retrieved context.
  • Example: For “What is the company’s remote work policy?” it checks if claims (e.g., “three days remote”) match the policy document.
• Context Entity Recall: Verifies that critical entities from the reference answer are included in the retrieved content.
  • Example: For “Tell me about the Taj Mahal,” ensures entities like “Shah Jahan” and “Agra” are retrieved.
• Context Recall: Measures if all necessary details from a reference are retrieved.
  • Example: For “What are the main benefits of product X?” it ensures all benefits like “cost savings” and “improved efficiency” are included.
• Noise Sensitivity: Checks if the system ignores irrelevant information in the retrieved context.
  • Example: For “What is LIC known for?” it ensures the response focuses on LIC’s attributes, filtering out unrelated economic data.
  

​

Evaluator Library: 50+ Ready-to-Use and Tailorable Evaluators

• The library includes 50+ preconfigured evaluation templates tailored for a wide range of use cases, from quality checks to compliance and contextual alignment.
• These templates are organized into categories such as LLM Evaluator Ragas Evaluator and Function Evaluator checks to help you find the right tool for your needs quickly.

• Each preconfigured evaluator can be customized and configured to fit specific project requirements.
  Once tailored, evaluators can be added to your personal Library for reuse in Experiment and Deployment.

• Evaluators are designed to work effortlessly in both experiments and deployments, enabling consistent and reliable quality control across development stages.

​

Improved LLMs as a Judge

• When you configure the output type as a number, boolean, or other types, the LLM ensures the result adheres strictly to the defined format.
• Additionally, each output includes a clear, concise explanation to support transparency and reliability in the evaluation process.

• The LLM-as-a-Judge now seamlessly integrates into live deployments, enabling asynchronous quality evaluation with a configurable sample rate. This approach provides actionable insights without disrupting workflows, ensuring consistent quality and minimizing drift.
• The LLM-as-a-Judge also supports Guardrail Mode, actively enforcing quality standards by blocking inputs or outputs that fail to meet defined criteria.

​

Online Evaluators in Live Deployments

​

Why Are Evaluators Important?

• Quality and Safety: Evaluators mitigate LLM risks by monitoring and ensuring output quality and safety.
• Performance Insights: They provide clear feedback on updates, helping teams identify improvements or regressions and refine iterations effectively.

​

What Are Evaluators?

• **Sample Rate Option:**Run evaluations on a subset of calls (e.g., 10%) to balance coverage and efficiency.
• **Cost Management:**Reduces additional expenses associated with running evaluators, especially LLM-based ones.
• Latency Control: Minimizes added response time by limiting the number of calls evaluated.

​

Cache your LLM response

1. Cost Savings

2. Time Savings

3. Improved Consistency

​

Structured Outputs on Azure

📘 Read more about Structured Outputs in Using Response Format.

​

Claude 3.5 Haiku and Sonnet (new)

​

Role Based Access Control improvements

• Admin: Complete access across all platform features, including user management, workspace settings, model configurations, and billing.
• Developer: Provides technical access to all essential features, including the ability to deploy changes within Deployments. Developers cannot manage models, billing, or users.
• Researcher: Offers access to Playgrounds, Experiments, and Deployments for research purposes. Researchers can review and annotate logs but cannot modify or deploy changes.

📘 Read more on how to set up permissions in our documentation

​

Attach files to a Deployment

📘 We support pdf, txt, docx, csv, xls - 10mb max

🚧 Be aware that we only extract the raw data, no images or objects. Also, encrypted files are not supported.

​

When should I attach a file and not use a Knowledge Base?

1. The need for full context understanding

2. Dynamic document context

3. Private or sensitive data

​

How to use it?

1. Upload your files

curl --location 'https://my.orq.ai/v2/files' \
--header 'Authorization: Bearer xxxxxx' \
--form 'purpose="retrieval"' \
--form 'file=@"/Users/cormick/Downloads/filename.pdf"'

2. Receive the _id

{
    "_id": "file_01JA5D27ZVW2N702Z0D3B1G8EK",
    "object_name": "files-api/workspaces/e747f6ac-19b0-47cd-8e79-0e1bf72b2a3e/retrieval/file_01JA5D27ZVW2N702Z0D3B1G8EK.vnd.openxmlformats-officedocument.spreadsheetml.sheet",
    "purpose": "retrieval",
    "file_name": "file_01JA5D27ZVW2N702Z0D3B1G8EK.vnd.openxmlformats-officedocument.spreadsheetml.sheet",
    "bytes": 5295,
    "created": "2024-10-14T11:36:54.189Z"
}

3. Send the file _id in addition to your normal Orq API call

curl --location 'https://my.orq.ai/v2/deployments/invoke' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer xxxxx' \
--data '{
    "key": "deployment_key",
    "messages": [
        {
            "role": "user",
            "content": ""
        }
    ],
    "file_ids": [
        "file_01JA5D27ZVW2N702Z0D3B1G8EK"
    ]
}'

​

Log Filtering

• Optimize Prompts: Filter logs with low ratings to pinpoint where your prompts need improvement and refine them for better outcomes.
• Troubleshoot Issues: Filter logs with error statuses to identify what went wrong and address any issues in your setup.
• Monitor Fallbacks: Filter logs that triggered a fallback model to see where the primary model didn’t perform as expected.

📘 Read more on our documentation page: Filters

​

Prompt Snippets

​

Using a Prompt Snippet

​

Configuring a Prompt Snippet

​

Version control

❗️ Please note that if you modify the snippet, all deployments containing that snippet will be automatically updated

📘 Read more on our documentation: Prompt.

​

Output Masking

​

Enabling Output Masking

📘 Read more on how to set up Output Masking in our Deployment Security and Privacy documentation

​

Workspace usage metrics

​

Structured outputs

​

Why should you use Structured Outputs?

• Reliable format: You don’t have to check or fix responses using the wrong format anymore.
• Clear refusals: When the model refuses something for safety reasons, it’s now easily detectable programmatically.
• Easier prompts: You don’t need to prompt the model to adhere to the schema anymore.

​

Using Structured Outputs

1. reponse_format (recommended) - In the dropdown menu (see GIF below), you can select the JSON pill and specify your schema.
2. Function Calling - In the tool, you can toggle the functionality of the structured output on/off.

• gpt-4o 2024-08-06 and later
• gpt-4o-mini 2024-07-18 and later

🚧 Reminder: To use Structured Outputs, all fields must be specified as required.

​

Vertex AI custom integration

​

Improved authentication

🚧 We removed the email and password login combination to enhance the security of our platform.

1. Magic Link Sign-In: Enter your email address, and we’ll send you a login link with a code for easy access. This means that you can also set up Two-Factor Authentication (2FA) or Multi-Factor Authentication (MFA) for added security.
2. Google Single Sign-On (SSO): Seamlessly log in using your Google account.
3. Microsoft Single Sign-On (SSO): Use your Microsoft account for quick and secure access.

​

New GoogleAI provider and Gemini models

📘 Google AI allows you to set up your own API keys, whereas VertexAI does not give you this option.

​

Why should I use Google AI instead of Vertex AI?

• Gemini-1.5-flash-8b-exp-0827
• Gemini-1.5-flash-exp-0827
• Gemini-1.5-pro-exp-0801
• Gemini-1.5-pro-exp-0827

​

Export Experiment Results

• Choose your file type: Select from CSV, JSON, or JSONL formats based on your specific requirements.
• Receive your data: After selecting your preferred format, your Experiment results will be compiled and sent to you via email, ready for further analysis.

• Leverage the exported data for custom visualizations, integrations, or further analysis in your preferred tools.
• Utilize the output of LLM generations in various formats for downstream tasks or additional finetuning.

​

Knowledge Bases & RAG

​

Why use a Knowledge Base?

​

Use cases

1. Reduced Hallucination: By relying on a well-structured knowledge base, the likelihood of the LLM generating incorrect or fabricated information (hallucinations) is significantly decreased.
2. Specific Context: A knowledge base allows for the inclusion of domain-specific or company-specific information, ensuring that the responses generated by the LLM are more aligned with the intended context.
3. Up-to-Date Information: Unlike static models, a knowledge base can be continuously updated with the latest information, providing the LLM with current and accurate data.

📘 Read more about Knowledge Bases in our Knowledge Base docs

​

Webhooks

• Automated data ingestion and processing: feeding real-time data from sources into Orq.
• Automated workflow orchestration: triggering specific deployments based on events.
• Logging: the ability to store the requests (input and output) back into your system.
• Monitoring and alerting: set up webhooks to alert relevant stakeholders about changes in model performance.

📘 Read more about webhooks in our docs.

​

Contact analytics

• Track Individual Users: Monitor costs and error rates for timely support.
• Track Internal Projects: Ensure projects stay within budget and meet goals.
• Track Client Projects: Get clear metrics per client projects.
• Track Specific Teams: Analyze team performance and optimize resource allocation.

📘 Read more on how to set up contact analytics in our docs

​

Onboard private models on Azure

​

Debug more effectively

​

Provider_response added in the SDK

{
  "id": "01J3MZHS6KFEWE7WBSAG0JP25X",
  "created": "2024-07-25T12:58:41.235Z",
  "object": "image",
  "model": "leonard-vision-xl",
  "provider": "leonardoai",
  "is_final": true,
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "url": "https://cdn.leonardo.ai/users/3bf1a64b-57f6-499b-907d-8b1286c1aa95/generations/dee000c8-7b6d-4d8c-82b4-d1992f8adce2/Default_a_tree_0.jpg"
      },
      "finish_reason": "stop"
    }
  ],
  "provider_response": {
    "generations_by_pk": {
      "generated_images": [
        {
          "url": "https://cdn.leonardo.ai/users/3bf1a64b-57f6-499b-907d-8b1286c1aa95/generations/dee000c8-7b6d-4d8c-82b4-d1992f8adce2/Default_a_tree_0.jpg",
          "nsfw": false,
          "id": "f47939ed-0841-48fa-b763-50e2e5b02e9a",
          "likeCount": 0,
          "motionMP4URL": null,
          "generated_image_variation_generics": []
        }
      ],
      "modelId": "5c232a9e-9061-4777-980a-ddc8e65647c6",
      "motion": null,
      "motionModel": null,
      "motionStrength": null,
      "prompt": "a tree",
      "negativePrompt": "",
      "imageHeight": 512,
      "imageToVideo": null,
      "imageWidth": 512,
      "inferenceSteps": 15,
      "seed": 4135326353,
      "public": false,
      "scheduler": "EULER_DISCRETE",
      "sdVersion": "SDXL_0_9",
      "status": "COMPLETE",
      "presetStyle": "CINEMATIC",
      "initStrength": null,
      "guidanceScale": null,
      "id": "dee000c8-7b6d-4d8c-82b4-d1992f8adce2",
      "createdAt": "2024-07-25T12:58:43.034",
      "promptMagic": false,
      "promptMagicVersion": null,
      "promptMagicStrength": null,
      "photoReal": false,
      "photoRealStrength": null,
      "fantasyAvatar": null,
      "generation_elements": []
    }
  }
}

​

GPT-4o mini

• Significantly smarter than GPT-3.5 Turbo, but not as capable as GPT-4o (see benchmark).
• 60% cheaper compared to GPT-3.5 Turbo.
• It’s a vision model, meaning it can interpret images.
• Context window is as big as GPT-4o (128k).

​

Claude 3.5 Sonnet

​

Prompt library

​

Llama 3 on Perplexity

• Generate up-to-date responses
• Retrieve dynamic data about the latest news etc.
• Understand real-world context better

​

Multi API key selector

​

Deployment test run

​

Groq custom integration

​

Keyboard shortcuts

​

New version pop-up

​

Tools workflow

​

New domain for ORQ.ai

🚧 If you’re using my.orquesta.dev please start using my.orq.ai

​

Version Controlled Deployments

​

Deployment Routing

​

Prompt generator

​

Tools in Experiments

​

Breadcrumb navigation

​

Experiment aggregate metrics

​

Set up your Tools in Resources

​

Google Gemini Pro 1.5

• 1 million context window! - Allowing it to understand a much greater context than before.
• Complex multimodality reasoning - It’s able to understand video, audio, code, image, and text.

​

Vision models are now available

• Claude-3-haiku
• Claude-3-sonnet
• Claude-3-opus
• GPT-4-turbo-vision
• GPT-4-vision-preview
• Gemini-pro-vision

​

New Cohere models - Command R & Command R+

• High precision on retrieval augmented generation (RAG) and tool use tasks (function calling)
• Low latency and high throughput
• Large context window 128,000-tokens
• Multilingual capabilities across 10 key languages, with pre-training data for 13 additional languages

​

Prompt Studio V2

​

Anthropic Custom Integration

​

AWS Bedrock Custom Integration

​

New SDK’s with breaking changes!

🚧 We recently transitioned our SDK's from@orquesta to @orq-ai across all platforms to align with our branding and focus on AI. This includes some breaking changes!

​

Python SDK

​

Installation

pip install orquesta-sdk

pip install orq-ai-sdk

​

Usage

from orquesta_sdk import Orquesta, OrquestaClientOptions

api_key = os.environ.get("ORQUESTA_API_KEY", "__API_KEY__")

options = OrquestaClientOptions(
    api_key=api_key,
    environment="production"
)

client = Orquesta(options)

import os

from orq_ai_sdk import OrqAI

client = OrqAI(
   api_key=os.environ.get("ORQ_API_KEY", "__API_KEY__"),
   environment="production"
)

​

Node SDK

​

Installation

npm install @orquesta/node

yarn add @orquesta/node

npm install @orq-ai/node

yarn add @orq-ai/node

​

Usage

import { createClient } from '@orquesta/node';

const client = createClient({
  apiKey: 'orquesta-api-key',
  environment: 'production',
});

import { createClient } from '@orq-ai/node';

const client = createClient({
  apiKey: 'orquesta-api-key',
  environment: 'production',
});

​

Playground V2

​

Claude 3 - Haiku, Sonnet and Opus are now available

​

Hyperlinking

1. Navigate towards the Logs tab
2. Click on the log that has the configuration that you want to iterate on
3. Click on the tree-dotted menu in the upper right corner
4. Click open in Playground
5. Click on ‘View Playground’
6. You’ll be redirected the Playground and see the exact same configuration you are using in your Deployment

​

Use Groq as your LLM provider

​

Make your own custom LLM Evaluator

​

Use LLM as a Reference in Experiments

​

New Providers & Models: Mistral large, Perplexity, Gemma 7b and other models are added

​

Import from Resources in Experiments

​

Resource management

​

Evaluators are available on our Platform

​

Python and Node SDK’s improvements

​

Support for messages in the invoke and stream functionalities

const deployment = await client.deployments.invoke({
  key: 'customer_service',
  messages: [
    {
      role: 'user',
      message:
        'A customer is asking about the latest software update features. Generate a detailed and informative response highlighting the key new features and improvements in the latest update.',
    },
  ],
  context: { environments: 'production', country: 'NLD' },
  inputs: { firstname: 'John', city: 'New York' },
  metadata: { customer_id: 'Qwtqwty90281' },
});

console.log(deployment?.choices[0].message.content);

1. Enhanced Contextual Clarity: The use of chat history empowers the model to preserve the context throughout a conversation. This feature ensures that each response is not only coherent but also directly relevant, as the model can draw upon past dialogues to fully grasp the nuances of the current inquiry or discussion topic.
2. Streamlined Conversation Flow: Chat history is instrumental in maintaining a consistent and logical flow in conversations. This prevents the occurrence of repetitive or conflicting responses, mirroring the natural progression of human dialogues and maintaining conversational integrity.
3. Tailored User Interactions: With access to previous interactions, chat history allows the model to customize its responses according to individual user preferences and historical queries. This level of personalization significantly boosts user engagement and satisfaction, leading to more effective and enjoyable communication experiences.

​

Support for messages and choices to add metrics

deployment.addMetrics({
  messages: [
    {
      role: 'user',
      message:
        'A customer is asking about the latest software update features. Generate a detailed and informative response highlighting the key new features and improvements in the latest update.',
    },
  ],
  choices: [
    {
      index: 0,
      finish_reason: 'stop',
      message: {
        role: 'assistant',
        content:
          "Dear customer: Thank you for your interest in our latest software update! We're excited to share with you the new features and improvements we've rolled out. Here's what you can look forward to in this update",
      },
    },
  ],
});

​

Support for OpenAI system fingerprint

​

Setup your own API key with the new AnyScale integration.

​

Mass Experimentation

​

Python and Node SDK’s now support multimodality

​

Function calling supported in the Playground

​

Billable transactions are now shown in the log details

​

Gemini, Google’s most capable model, now available on Orquesta

​

Log and metrics enhancement

​

Chat history now available in the playground

​

Support for Dall-E models

​

orq.ai Integrations

• OpenAI
• Azure
• Cohere

​

Reset models default parameter in a Deployment

​

AWS Bedrock models available

• anthropic/claude-2
• anthropic/claude
• anthropic/claude-instant
• a21labs/jurassic-2-ultra
• a21labs/jurassic-2-mid
• amazon/titan-text-lite
• amazon/titan-text-express
• cohere/command
• anthropic/claude-2.1
• cohere/command-light