> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orq.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Updates from 2024

> Orq.ai product updates announced in 2024.

**December 16th, 2024**

## Online Guardrails in Live Deployments

You can now configure **Guardrails** after you have added them to your **Library** directly in **Deployments > Settings** for both input and output, giving you full control over [Deployment responses](https://docs.orq.ai/ai-studio/ai-engineering/deployments.md#evaluators-and-guardrails)

* **Guardrails:** Block unacceptable inputs or outputs based on predefined criteria. If a guardrail is triggered, it can either retry the request or activate a fallback model, if configured.

## **What Are Guardrails?**

Guardrails enforce boundaries by setting thresholds to block inputs or outputs that fail to meet predefined criteria. When triggered, guardrails can stop processing or activate fallback mechanisms like retries or alternative models.

* **Threshold Configuration:** Set customizable thresholds based on your specific use case to block inputs or outputs that fail to meet predefined criteria.
* **Input and Output Guardrails:** Apply rules to both incoming data and generated outputs, ensuring comprehensive quality control at every stage.
* **Synchronous Operation:** Guardrails run synchronously and do not introduce additional latency to your workflows.
* **Fallback Mechanisms:** When triggered, guardrails can stop processing or activate fallback options such as retries or alternative models to maintain continuity.

**December 13th, 2024**

## **Why Are Evals and Guardrails Important?**

With this release, you can gain full control over your AI applications, enhancing the reliability, safety, and quality of AI-generated outputs with evaluators and guardrails. While LLMs are incredibly powerful, they are also prone to errors such as incorrect information, biased content, or misaligned outputs—a risk famously illustrated by a chatbot that mistakenly agreed to sell a car for \$1.

The value of evaluators is twofold:

1. First, they address the inherent risks of LLMs by monitoring and ensuring output quality and safety.
2. Second, they help teams overcome one of AI development’s biggest challenges: understanding how updates impact performance. Without clear insights, the development loop can break, turning iteration into guesswork rather than a structured engineering process. Evaluators help you understand whether an update is an improvement or a regression and quickly drill down into good / bad examples.

## **What Are Evaluators?**

Evaluators assess and score outputs either against a reference or directly as they are generated by the model. They play a critical role in monitoring and improving AI quality, enabling you to evaluate performance continuously without disrupting operations. Evaluators come in several forms:

**Classic Evaluators:** These rely on references and established metrics, such as cosine similarity or BLEU scores, to compare outputs to an expected result. They are ideal for structured, deterministic tasks like translation or summarization.

* Example: Use cosine similarity to measure how closely an LLM-generated summary matches the reference text.

**LLM-as-a-Judge:** This approach uses language models themselves to evaluate outputs in real time, without requiring reference texts. By leveraging tailored prompts, LLMs can assess dimensions such as tone, accuracy, and task relevance, offering more flexibility for dynamic and creative use cases.

* Example: Evaluate whether a chatbot’s response aligns with the intended tone and emotional context without comparing it to a specific reference.

We provide four specialized types of evaluators, each tailored to specific evaluation needs:

**RAGAS Evaluators**\
Designed to evaluate retrieval-augmented generation (RAG) workflows, these evaluators measure context relevance, faithfulness, recall, and robustness. They ensure responses derived from external knowledge bases are accurate and focused.

* Example: Verify that a chatbot retrieving insurance policy details provides relevant and accurate information while excluding unrelated content.

**Function Evaluators**\
Rule-based tools for assessing measurable aspects of text, such as formatting, length, or keyword presence. These provide binary, deterministic results for compliance and validation.

* Example: Check if a response includes “return policy” or stays within a 200-character limit for customer service use cases.

**HTTP Evaluators**\
Allow external API calls to evaluate outputs using custom or third-party systems, enabling tailored checks for domain-specific requirements.

* Example: Call a compliance-check API to validate that AI-generated responses meet industry-specific guidelines.

**JSON Evaluators**\
Validate JSON outputs against predefined schemas, ensuring all required fields, correct data types, and structures are present.

* Example: Confirm a JSON file for user registration includes fields like “username,” “age,” and “email” in the correct format.

## **What Are Guardrails?**

**Guardrails**, enforce strict boundaries to block unacceptable inputs or outputs entirely. These constraints ensure the system adheres to rules like regulatory compliance, ethical standards, or content safety. The motives to implement guardrails can be summarized in four categories:

* **Robustness and Security.** Guardrails are of huge importance in the aspect of security and robustness of an LLM-based application. This includes vulnerabilities such as prompt injections, jail-breaking, data leakage, handling illegible or obfuscated content.
* **Information and Evidence.** Guardrails can be put in place for fact checking, detecting hallucinations, making sure the response does not contain incorrect or misrepresented information, does not consider irrelevant information, the response is supported by evidence.
* **Ethics and Safety.** Making sure the response does not lead to harmful consequences either directly or indirectly. A response should not exhibit problematic social biases or share protected or sensitive information, take into account copyright and plagiarism law, use 3rd party content by appropriately citing it, etc.
* **Tool-specific functionalities.** Making sure the responses are on-topic, of proper extensiveness, written in the suitable tone, use specified terminology, etc.

**December 13th, 2024**

## HTTP and JSON Evaluators and Guardrails

You can now add HTTP and JSON Evaluators and Guardrails under the Evaluator tab and add them to your [Deployment](https://docs.orq.ai/ai-studio/ai-engineering/deployments.md) or [Experiment](https://docs.orq.ai/ai-studio/optimize/experiments.md).

### **HTTP Evaluators and Guardrails**

* **Purpose:** Enable evaluation using external APIs to perform checks like compliance, custom logic, or business-specific validations.
* **How It Works:** You define the URL, headers, and payload for [HTTP Requests](https://docs.orq.ai/ai-studio/optimize/evaluators.md), and the evaluator expects either a boolean or numerical response in the format:

```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{ "type": "boolean", "value": true }
```

```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{ "type": "number", "value": 1 }
```

* **Use Case Example:** An HTTP Evaluator can validate content against regulatory guidelines by routing a compliance check to your API, returning a boolean to accept or block the output. If you want to integrate an HTTP Eval into your set up, you can find more information in [HTTP Evaluators](https://docs.orq.ai/ai-studio/optimize/evaluators.md).
* **Guardrails:** Configurable to accept or deny calls based on the returned value, providing dynamic control in experiments and deployments.

### **JSON Evaluators and Guardrails**

* **Purpose:** Validate input or output payloads during [Experiments](https://docs.orq.ai/ai-studio/optimize/experiments.md) or [Deployments](https://docs.orq.ai/ai-studio/ai-engineering/deployments.md) using a predefined JSON Schema.
* **How It Works:** Enter a JSON Schema specifying required fields and their expected types. For example:

```
{
  "name": "ReceiptDataExtraction",
  "strict": true,
  "schema": {
    "type": "object",
    "properties": {
      "Date": {
        "type": "string",
        "description": "Transaction date (YYYY-MM-DD)."
      },
      "Vendor": {
        "type": "string",
        "description": "Name of the vendor."
      },
      "Amount": {
        "type": "number",
        "description": "Total transaction amount."
      }
    },
    "required": ["Date", "Vendor", "Amount"],
    "additionalProperties": false
  }
}
```

* **Use Case Example:** A JSON Evaluator can ensure that generated content includes mandatory fields like "title" and "length," blocking invalid payloads. For a more extensive explanation on how to set this up in your Deployment, you can go [here](https://docs.orq.ai/ai-studio/optimize/evaluators.md).
* **Guardrails:** Toggle guardrails to block outputs that don’t adhere to the schema, ensuring compliance during Deployments or Experiments.

**December 13th, 2024**

## Master Your RAG with RAGAS Evals

The [Ragas Evaluators](https://docs.orq.ai/ai-studio/marketplace.md#ragas-evaluators) are now available, providing specialized tools to evaluate retrieval-augmented generation (RAG) workflows. These evaluators make it easy to set up quality checks when integrating a [Knowledge Base](https://docs.orq.ai/ai-studio/ai-engineering/knowledge-bases.md) into a RAG system and can be used in [Experiments](https://docs.orq.ai/ai-studio/optimize/experiments.md) and [Deployment](https://docs.orq.ai/ai-studio/ai-engineering/deployments.md) to ensure responses are accurate, relevant, and safe.

**Key Features**

* **Out-of-the-Box Functionality:** RAGAS Evaluators are ready to use and *cannot be reconfigured*, offering a consistent evaluation framework.
* **Reference-Based Scoring:** Some evaluators require a reference to calculate metrics like accuracy or faithfulness.
* **Scoring Scale:** Evaluations return a score between 0 and 1, with higher scores indicating better performance (e.g., higher relevance or faithfulness).

**Included Evaluators**

* **Context Precision:** Assesses how well retrieved chunks align with the user’s query.
  * Example: Ensures chunks about “home insurance” are prioritized when the query asks about coverage, filtering out irrelevant topics like auto insurance.
* **Response Relevancy:** Evaluates how directly the generated answer addresses the query.
  * Example: For “What are the fees for international transfers?” it ensures the answer is concise and focused on the fees without unrelated details.
* **Faithfulness:** Ensures the response is factually consistent with the retrieved context.
  * Example: For “What is the company’s remote work policy?” it checks if claims (e.g., “three days remote”) match the policy document.
* **Context Entity Recall:** Verifies that critical entities from the reference answer are included in the retrieved content.
  * Example: For “Tell me about the Taj Mahal,” ensures entities like “Shah Jahan” and “Agra” are retrieved.
* **Context Recall:** Measures if all necessary details from a reference are retrieved.
  * Example: For “What are the main benefits of product X?” it ensures all benefits like “cost savings” and “improved efficiency” are included.
* **Noise Sensitivity:** Checks if the system ignores irrelevant information in the retrieved context.

  * Example: For “What is LIC known for?” it ensures the response focuses on LIC’s attributes, filtering out unrelated economic data.

  ![RAGAS evaluators configured in a Deployment showing context precision and faithfulness scores.](https://mintcdn.com/orqai/4YNqGRNpuZNyo0_T/images/changelog-ragas-evaluators-deployment.png?fit=max\&auto=format\&n=4YNqGRNpuZNyo0_T\&q=85\&s=8a570edb620f9e1ea6aa48070760db2c)

**December 13th, 2024**

## Evaluator Library: 50+ Ready-to-Use and Tailorable Evaluators

Introducing the new Evaluator Library:

**Key Features:**

**1. Gallery of Preconfigured Evaluators:**

* The library includes 50+ preconfigured evaluation templates tailored for a wide range of use cases, from quality checks to compliance and contextual alignment.
* These templates are organized into categories such as [LLM Evaluator](https://docs.orq.ai/ai-studio/marketplace.md#llm-evaluators), [Ragas Evaluator](https://docs.orq.ai/ai-studio/marketplace.md#ragas-evaluators), and [Function Evaluator](https://docs.orq.ai/ai-studio/marketplace.md#function-evaluators) checks to help you find the right tool for your needs quickly.

**2. Customizable and Configurable Evaluators:**

* Each preconfigured evaluator can be customized and configured to fit specific project requirements.\
  Once tailored, evaluators can be added to your personal **Library** for reuse in [Experiment](https://docs.orq.ai/ai-studio/optimize/experiments.md) and [Deployment](https://docs.orq.ai/ai-studio/ai-engineering/deployments.md).

**3. Seamless Integration:**

* Evaluators are designed to work effortlessly in both experiments and deployments, enabling consistent and reliable quality control across development stages.

  ![Evaluator Library showing a gallery of 50+ preconfigured evaluator templates organized by category.](https://mintcdn.com/orqai/4YNqGRNpuZNyo0_T/images/changelog-evaluator-library.png?fit=max\&auto=format\&n=4YNqGRNpuZNyo0_T\&q=85\&s=3e51fc0384e1a4a3620fdf5eabda9945)

**December 13th, 2024**

## Improved LLMs as a Judge

**LLM-as-a-Judge Enhancements:**\
We’ve significantly improved our existing [LLM Evaluator](https://docs.orq.ai/ai-studio/marketplace.md#llm-evaluators) feature to provide more robust evaluation capabilities and enforce type-safe outputs.

**1. Type-Safe Output:**

* When you configure the output type as a number, boolean, or other types, the LLM ensures the result adheres strictly to the defined format.
* Additionally, each output includes a clear, concise explanation to support transparency and reliability in the evaluation process.

  ![LLM-as-a-Judge configuration with type-safe numeric or boolean output and a scored explanation.](https://mintcdn.com/orqai/4YNqGRNpuZNyo0_T/images/changelog-llm-judge-config.png?fit=max\&auto=format\&n=4YNqGRNpuZNyo0_T\&q=85\&s=2eb9656837aeb44da741e27766b6f5e7)

**2. Extended Functionality:**

* The LLM-as-a-Judge now seamlessly integrates into live deployments, enabling asynchronous quality evaluation with a configurable sample rate. This approach provides actionable insights without disrupting workflows, ensuring consistent quality and minimizing drift.
* The LLM-as-a-Judge also supports Guardrail Mode, actively enforcing quality standards by blocking inputs or outputs that fail to meet defined criteria.

  ![LLM-as-a-Judge integrated into a live Deployment in Guardrail Mode for blocking outputs that fail quality criteria.](https://mintcdn.com/orqai/4YNqGRNpuZNyo0_T/images/changelog-llm-judge-guardrail-mode.png?fit=max\&auto=format\&n=4YNqGRNpuZNyo0_T\&q=85\&s=c068528437255f23323d29805d83c825)

**November 30th, 2024**

## Online Evaluators in Live Deployments

You can now configure **Evaluators** after you have added them to your **Library** directly in **Deployments > Settings** for both input and output, giving you full control over [Deployments](https://docs.orq.ai/ai-studio/ai-engineering/deployments.md).

### **Why Are Evaluators Important?**

* **Quality and Safety:** Evaluators mitigate LLM risks by monitoring and ensuring output quality and safety.
* **Performance Insights:** They provide clear feedback on updates, helping teams identify improvements or regressions and refine iterations effectively.

## **What Are Evaluators?**

Evaluators are tools designed to assess the quality, relevance, and safety of AI inputs or outputs, ensuring reliable and effective system performance. Off the shelf, we offer several options, including [LLM Evaluator](https://docs.orq.ai/ai-studio/marketplace.md#llm-evaluators), [Ragas Evaluator](https://docs.orq.ai/ai-studio/marketplace.md#ragas-evaluators), [Function Evaluator](https://docs.orq.ai/ai-studio/marketplace.md#function-evaluators) and [HTTP Evaluators](https://docs.orq.ai/ai-studio/optimize/evaluators.md). Additionally, the evaluators allow for the following configuration:

* **Sample Rate Option:** Run evaluations on a subset of calls (e.g., 10%) to balance coverage and efficiency.
* **Cost Management:** Reduces additional expenses associated with running evaluators, especially LLM-based ones.
* **Latency Control:** Minimizes added response time by limiting the number of calls evaluated.

If you want to set up the evals in your own deployment, you can find more information [here](https://docs.orq.ai/ai-studio/ai-engineering/deployments.md#evaluators-and-guardrails).

<video src="https://mintcdn.com/orqai/1oqtKOJj_dB5Fg2y/videos/newevals.mp4?fit=max&auto=format&n=1oqtKOJj_dB5Fg2y&q=85&s=3b3383185f1fe8771c6a2c2bf95b9972" autoPlay muted loop playsInline>
  Animation showing how to configure online evaluators directly in the Settings of a live Deployment.
</video>

**November 25th, 2024**

## Cache your LLM response

Cache and reuse your LLM outputs for near-instant responses, reduced costs, and consistent results.

**What is Caching?**

With this release, you can now enable caching for your LLM calls. Caching works by storing both the input and the generated output for a configurable period (TTL – Time-to-Live). When an exact request is made during this time, the cached output is returned instantly, bypassing the need for the LLM to generate a new response.

**Why Enable Caching?**

1. Cost Savings

By reusing previously generated responses, caching reduces the number of LLM calls, which can significantly cut down on usage costs.

2. Time Savings

Cached responses are nearly instant, offering a much faster response time compared to generating a new output from the LLM.

3. Improved Consistency

Caching mitigates the inherent variability of LLM responses by ensuring that identical inputs produce the same outputs during the cache period.

**How to set up Caching:**

Go to your Deployment > settings > enable caching > configure TTL

**How to check the Cache status:**

Logs > select specific log > request tab > caching status

In here you will see if the cache was hit or not and the configured TTL (Time-to-Live) in seconds.

**November 22nd, 2024**

## Structured Outputs on Azure

Ensure the model’s responses always stick to your JSON Schema. No more missing fields or incorrect values, making your AI applications more reliable and robust.

In addition to the Open AI provided models, you can now use Structured Outputs on Azure hosted models.

Using Structured Outputs is the go-to method if you want to ensure that the model is adhering to your JSON Schema.

> 📘 Read more about Structured Outputs in [Using Response Format](https://docs.orq.ai/ai-gateway/features/structured-outputs.md).

**November 7th, 2024**

## Claude 3.5 Haiku and Sonnet (new)

Introducing the upgraded Claude 3.5 Sonnet and the new Claude 3.5 Haiku. The enhanced Claude 3.5 Sonnet offers comprehensive improvements, especially in coding and tool use, while maintaining the speed and pricing of the previous model. These upgrades make it an excellent choice for complex, multi-step development and planning tasks.

For those prioritizing speed, Claude 3.5 Haiku combines quick response times with strong overall performance. It’s ideal for user-facing applications and specialized tasks, managing data-rich inputs like personalized recommendations and real-time decision-making with ease.

Note: Claude 3.5 Haiku doesn't support vision just yet.

![Claude 3.5 Haiku and the updated Claude 3.5 Sonnet models now available in the Orq.ai AI Router.](https://mintcdn.com/orqai/kH7ELCU0voLJRri0/images/43bd67a1c88d6012e78a6d90ddee29e66f781a1cb7de3552ee6feac2671a09be-image.png?fit=max\&auto=format\&n=kH7ELCU0voLJRri0\&q=85\&s=6e2157ef07a78e40c88b8fd8f0471b80)

**November 4th, 2024**

## Role Based Access Control improvements

With the improvements to role-based access control (RBAC), it is much easier to assign and change workspace roles in Orq.

Orq now offers three distinct workspace roles:

* **Admin**: Complete access across all platform features, including user management, workspace settings, model configurations, and billing.
* **Developer**: Provides technical access to all essential features, including the ability to deploy changes within Deployments. Developers cannot manage models, billing, or users.
* **Researcher**: Offers access to Playgrounds, Experiments, and Deployments for research purposes. Researchers can review and annotate logs but cannot modify or deploy changes.

> 📘 Read more on how to set up permissions in [our documentation](https://docs.orq.ai/ai-studio/organization/members-teams.md)

**October 14th, 2024**

## Attach files to a Deployment

One common request we hear is whether it’s possible to attach files to an LLM — for example, to extract data from a PDF or engage with its content. With this update, you can now do just that.

> 📘 We support pdf, txt, docx, csv, xls - 10mb max

The content of the files will be incorporated into the initial system or user message. We recommend adjusting your prompt as you would when using variables in a prompt. The added content will be highlighted in green. See the example below:

> 🚧 Be aware that we only extract the raw data, no images or objects. Also, encrypted files are not supported.

## When should I attach a file and not use a Knowledge Base?

1. **The need for full context understanding**

RAG retrieves relevant chunks, which works for focused queries but falls short for tasks like summarization that require full-document understanding. Attaching files gives the LLM access to the entire document, ensuring it has the complete context.

For example, when summarizing reports, legal cases, or research papers, the LLM needs to process the full document to capture key details and connections that partial text retrieval can’t provide. Full context access leads to better comprehension and more accurate outputs, particularly for tasks requiring a holistic view, such as summarization and detailed analysis.

2. **Dynamic document context**

Unlike a static knowledge base, attached files can provide ad-hoc, context-specific documents for one-time or immediate use without the need for integration into a broader knowledge repository.

When a user is dealing with unique documents—such as one-off reports, meeting notes, or specific contracts—they can attach these files directly to a deployment. The LLM can instantly use these documents to provide answers or insights. This feature is especially useful for situations where time-sensitive or project-specific documents need to be used on the fly, giving flexibility to quickly incorporate new, temporary knowledge without modifying or updating the knowledge base.

3. **Private or sensitive data**

Due to privacy concerns, confidential or sensitive files (e.g., contracts and medical records) may not be suitable for a general knowledge base. Attaching files directly allows secure, temporary interaction with this data.

## How to use it?

To attach one or more files to a Deployment, follow these steps:

1. Upload your files

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --location 'https://my.orq.ai/v2/files' \
--header 'Authorization: Bearer xxxxxx' \
--form 'purpose="retrieval"' \
--form 'file=@"/Users/cormick/Downloads/filename.pdf"'
```

2. Receive the `_id`

```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
    "_id": "file_01JA5D27ZVW2N702Z0D3B1G8EK",
    "object_name": "files-api/workspaces/e747f6ac-19b0-47cd-8e79-0e1bf72b2a3e/retrieval/file_01JA5D27ZVW2N702Z0D3B1G8EK.vnd.openxmlformats-officedocument.spreadsheetml.sheet",
    "purpose": "retrieval",
    "file_name": "file_01JA5D27ZVW2N702Z0D3B1G8EK.vnd.openxmlformats-officedocument.spreadsheetml.sheet",
    "bytes": 5295,
    "created": "2024-10-14T11:36:54.189Z"
}
```

3. Send the file `_id` in addition to your normal Orq API call

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --location 'https://my.orq.ai/v2/deployments/invoke' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json' \
--header 'Authorization: Bearer xxxxx' \
--data '{
    "key": "deployment_key",
    "messages": [
        {
            "role": "user",
            "content": ""
        }
    ],
    "file_ids": [
        "file_01JA5D27ZVW2N702Z0D3B1G8EK"
    ]
}'
```

**October 11th, 2024**

## Log Filtering

The ‘Log Filters’ feature is now available, following high demand from our users. Monitoring logs is essential for tracking performance and identifying areas for improvement, and with this new feature, it’s now easier to find the exact data you need to make informed decisions.

With Log Filters, you can quickly sort through large sets of logs and focus on the most relevant information for your workflow. Here are some key use cases:

* **Optimize Prompts**: Filter logs with low ratings to pinpoint where your prompts need improvement and refine them for better outcomes.
* **Troubleshoot Issues**: Filter logs with error statuses to identify what went wrong and address any issues in your setup.
* **Monitor Fallbacks**: Filter logs that triggered a fallback model to see where the primary model didn’t perform as expected.

<video src="https://mintcdn.com/orqai/1oqtKOJj_dB5Fg2y/videos/log_filters.mp4?fit=max&auto=format&n=1oqtKOJj_dB5Fg2y&q=85&s=e9b107aa262b3aad568cf722a698a4fd" autoPlay muted loop playsInline>
  Animation showing the Log Filters feature for sorting logs by rating, status, or fallback trigger.
</video>

> 📘 Read more on our documentation page: [Filters](https://docs.orq.ai/ai-studio/ai-engineering/deployments.md#analytics-and-logs)

**October 11th, 2024**

## Prompt Snippets

Introducing Prompt Snippets, a feature designed to boost your efficiency by allowing you to reuse key parts of your prompt across multiple variants.

When working with multiple deployment variants that share common prompt elements, you no longer need to update each one individually. Instead, edit the snippet once, and all related variants will automatically reflect the change, saving you time and reducing manual effort.

## Using a Prompt Snippet

When you type `{{snippet.`, a list of all available snippets will appear. You can then select the snippet you’d like to use.

## Configuring a Prompt Snippet

Head to the Prompt Library to create and configure your Prompt Snippet.

### Version control

Prompt Snippets are fully version-controlled, making it easier to track changes and correct mistakes, especially in collaborative environments. Be aware that updates to snippets will automatically apply to all deployments using them, so ensure any changes are intentional and properly tested.

> ❗️ Please note that if you modify the snippet, all deployments containing that snippet will be automatically updated

> 📘 Read more on our documentation: [Prompt](https://docs.orq.ai/ai-studio/prompts/prompts.md).

**October 11th, 2024**

## Output Masking

We’ve added a new Output Masking feature to further strengthen security for the most sensitive use cases. With this feature, you can now mask the entire model’s output in a Deployment, providing an additional layer of data protection.

In addition to the existing capability to mask input variables—such as marking patient\_information as PII to prevent sensitive data from being stored on the Orq.ai platform—Output Masking extends this protection to the model’s complete output as well, ensuring that sensitive information remains secure throughout the entire process.

## Enabling Output Masking

Go to your Deployment > Settings > toggle Output Masking on.

> 📘 Read more on how to set up Output Masking in our [Deployment Security and Privacy documentation](https://docs.orq.ai/ai-studio/ai-engineering/deployments.md#security-and-privacy)

**October 11th, 2024**

## Workspace usage metrics

We’ve added a new feature to the Billing page called Workspace Usage. This tool provides users with a clear overview of how many API calls they’ve used out of their available limit within their current billing cycle. The usage tracking starts from the beginning of your subscription.

It’s important to note that **exceeding your API usage limit does not stop you from making further calls.** However, once you surpass the free API call limit, additional charges will apply for any extra usage.

**September 16th, 2024**

## Structured outputs

Structured Outputs is a feature that makes sure the model's responses always follow your provided JSON Schema. This means you don't have to worry about the model leaving out any required parts or making up incorrect values.

## Why should you use Structured Outputs?

* Reliable format: You don’t have to check or fix responses using the wrong format anymore.
* Clear refusals: When the model refuses something for safety reasons, it’s now easily detectable programmatically.
* Easier prompts: You don’t need to prompt the model to adhere to the schema anymore.

## Using Structured Outputs

There are two ways of using the Structured Outputs functionality:

1. **`response_format`** (recommended) - In the dropdown menu (see the video below), you can select the JSON pill and specify your schema.
2. **Function Calling** - In the tool, you can toggle the functionality of the structured output on/off.

Note: Structured Outputs are only available when using the following models:

* gpt-4o 2024-08-06 and later
* gpt-4o-mini 2024-07-18 and later

> 🚧 Reminder: To use Structured Outputs, all fields must be specified as required.

If you want to read more about Structured Outputs, please visit OpenAI's documentation [here](https://platform.openai.com/docs/guides/structured-outputs)

**September 16th, 2024**

## Vertex AI custom integration

Set up your custom API keys with the new Vertex AI integration.

![Vertex AI custom integration setup showing API key configuration in Orq.ai.](https://mintcdn.com/orqai/4YNqGRNpuZNyo0_T/images/changelog-vertex-ai-integration.png?fit=max\&auto=format\&n=4YNqGRNpuZNyo0_T\&q=85\&s=29004272377182140e063cbfe27a0718)

**September 16th, 2024**

## Improved authentication

> 🚧 We removed the email and password login combination to enhance the security of our platform.

To improve the security of our platform, you will be able to log in using the following methods:

1. **Magic Link Sign-In**: Enter your email address, and we’ll send you a login link with a code for easy access. This means that you can also set up Two-Factor Authentication (2FA) or Multi-Factor Authentication (MFA) for added security.
2. **Google Single Sign-On** (SSO): Seamlessly log in using your Google account.
3. **Microsoft Single Sign-On** (SSO): Use your Microsoft account for quick and secure access.

These changes aim to provide a more secure and convenient login experience.

**August 28th, 2024**

## New GoogleAI provider and Gemini models

With the introduction of **Google AI** as a new provider on the platform, the previous 'Google' provider is renamed to **Vertex AI**.

If you were using Google as a provider before, you don't have to do anything, you will only see a different provider name. The model will remain exactly the same.

> 📘 Google AI allows you to set up your own API keys, whereas Vertex AI does not give you this option.

## Why should I use Google AI instead of Vertex AI?

In addition to Google AI allowing you to set up your own API keys, they also offer their latest and experimental models. So, if you want to test out their newest models, Google AI is the way to go.

The new models that are available in Google AI:

* Gemini-1.5-flash-8b-exp-0827
* Gemini-1.5-flash-exp-0827
* Gemini-1.5-pro-exp-0801
* Gemini-1.5-pro-exp-0827

  ![New Google AI provider in the AI Router showing the latest Gemini experimental models available.](https://mintcdn.com/orqai/kH7ELCU0voLJRri0/images/1774e8291d985a3147736120c7aa1163b40b3b61422d5dd5275deb494d5cadd4-image.png?fit=max\&auto=format\&n=kH7ELCU0voLJRri0\&q=85\&s=221d7cb59a3f51c42fd1a3770ed55a84)

**August 28th, 2024**

## Export Experiment Results

To give data scientists greater flexibility, we now allow you to export your Experiment results directly from Orq.ai. This update provides the freedom to extend your analysis beyond the platform, enabling you to work with your data in your preferred tools and environments.

How it works:

* **Choose your file type**: Select from CSV, JSON, or JSONL formats based on your specific requirements.
* **Receive your data**: After selecting your preferred format, your Experiment results will be compiled and sent to you via email, ready for further analysis.

**Use cases**

* Leverage the exported data for custom visualizations, integrations, or further analysis in your preferred tools.
* Utilize the output of LLM generations in various formats for downstream tasks or additional finetuning.

**August 15th, 2024**

## Knowledge Bases & RAG

Start building even more reliable AI applications by equipping LLMs with accurate and relevant knowledge. This will greatly improve their knowledge and thus reduce the likelihood of hallucinations.

## Why use a Knowledge Base?

The primary purpose of a knowledge base and a Retrieval-Augmented Generation (RAG) is to provide a reliable source of information that an LLM can access. By querying a knowledge base, an LLM can retrieve relevant data to answer questions or solve problems more accurately. This integration ensures that the information provided is both relevant and precise, enhancing the overall effectiveness of the model.

## Use cases

1. **Reduced Hallucination**: By relying on a well-structured knowledge base, the likelihood of the LLM generating incorrect or fabricated information (hallucinations) is significantly decreased.
2. **Specific Context**: A knowledge base allows for the inclusion of domain-specific or company-specific information, ensuring that the responses generated by the LLM are more aligned with the intended context.
3. **Up-to-Date Information**: Unlike static models, a knowledge base can be continuously updated with the latest information, providing the LLM with current and accurate data.

> 📘 Read more about Knowledge Bases in our [Knowledge Base docs](https://docs.orq.ai/ai-studio/ai-engineering/knowledge-bases.md)

**August 1st, 2024**

## Webhooks

A much-requested feature, webhooks, is now added to the Orq.ai platform.

Webhooks allow you to **automate** a lot of things **in real-time**, including:

* Automated data ingestion and processing: feeding real-time data from sources into Orq.
* Automated workflow orchestration: triggering specific deployments based on events.
* Logging: the ability to store the requests (input and output) back into your system.
* Monitoring and alerting: set up webhooks to alert relevant stakeholders about changes in model performance.

In summary, with the new webhooks, you'll get much more flexibility in building your Gen AI application.

> 📘 Read more about webhooks in our [docs](https://docs.orq.ai/ai-studio/organization/webhooks.md).

**August 1st, 2024**

## Contact analytics

To gain **more insights and control over your LLM consumption**, we added **contact analytics**.

A contact is an entity that can be anything you'd like to track, such as an **individual, team**, **project**, or **client**.

Some use cases include:

* **Track Individual Users**: Monitor costs and error rates for timely support.
* **Track Internal Projects**: Ensure projects stay within budget and meet goals.
* **Track Client Projects:** Get clear metrics per client projects.
* **Track Specific Teams**: Analyze team performance and optimize resource allocation.

  ![Contact analytics dashboard showing cost and usage metrics per contact in Orq.ai.](https://mintcdn.com/orqai/eNeVsDsn_DMSI7MV/images/changelog-contacts-analytics.jpg?fit=max\&auto=format\&n=eNeVsDsn_DMSI7MV\&q=85\&s=4f8218b8c2e75abb45d38eabef594283)

> 📘 Read more on how to set up contact analytics in our [docs](https://docs.orq.ai/ai-studio/observability/identities.md)

**July 25th, 2024**

## Onboard private models on Azure

You can now add your own private and/or finetuned models to the **Orq.ai** **AI Router**.

As of now, you can only add models hosted on Azure. Soon, we'll also support the self-onboarding of private models through OpenAI and Vertex AI.

Use the interactive walkthrough below to see how the onboarding process works.

**July 25th, 2024**

## Debug more effectively

The **Debug** tab is back. Did your request fail, and do you want to look into the payload? Navigate towards the debug tab and find out what has been happening.

![The Debug tab showing the raw request and response payload for a failed LLM call.](https://mintcdn.com/orqai/kH7ELCU0voLJRri0/images/5562ddf-debug.jpg?fit=max\&auto=format\&n=kH7ELCU0voLJRri0\&q=85\&s=9498e16e7743c00469fc4beffb13a2dd)

**July 25th, 2024**

## Provider\_response added in the SDK

We did add the raw `provider_response` to our Python and Node SDK. This will give you the flexibility of retrieving model-specific output that we don't provide through our unified API.

See an example response below:

```json theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
  "id": "01J3MZHS6KFEWE7WBSAG0JP25X",
  "created": "2024-07-25T12:58:41.235Z",
  "object": "image",
  "model": "leonard-vision-xl",
  "provider": "leonardoai",
  "is_final": true,
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "url": "https://cdn.leonardo.ai/users/3bf1a64b-57f6-499b-907d-8b1286c1aa95/generations/dee000c8-7b6d-4d8c-82b4-d1992f8adce2/Default_a_tree_0.jpg"
      },
      "finish_reason": "stop"
    }
  ],
  "provider_response": {
    "generations_by_pk": {
      "generated_images": [
        {
          "url": "https://cdn.leonardo.ai/users/3bf1a64b-57f6-499b-907d-8b1286c1aa95/generations/dee000c8-7b6d-4d8c-82b4-d1992f8adce2/Default_a_tree_0.jpg",
          "nsfw": false,
          "id": "f47939ed-0841-48fa-b763-50e2e5b02e9a",
          "likeCount": 0,
          "motionMP4URL": null,
          "generated_image_variation_generics": []
        }
      ],
      "modelId": "5c232a9e-9061-4777-980a-ddc8e65647c6",
      "motion": null,
      "motionModel": null,
      "motionStrength": null,
      "prompt": "a tree",
      "negativePrompt": "",
      "imageHeight": 512,
      "imageToVideo": null,
      "imageWidth": 512,
      "inferenceSteps": 15,
      "seed": 4135326353,
      "public": false,
      "scheduler": "EULER_DISCRETE",
      "sdVersion": "SDXL_0_9",
      "status": "COMPLETE",
      "presetStyle": "CINEMATIC",
      "initStrength": null,
      "guidanceScale": null,
      "id": "dee000c8-7b6d-4d8c-82b4-d1992f8adce2",
      "createdAt": "2024-07-25T12:58:43.034",
      "promptMagic": false,
      "promptMagicVersion": null,
      "promptMagicStrength": null,
      "photoReal": false,
      "photoRealStrength": null,
      "fantasyAvatar": null,
      "generation_elements": []
    }
  }
}
```

**July 24th, 2024**

## GPT-4o mini

OpenAI has introduced GPT-4o mini, a smaller, more efficient, and cheaper model replacing GPT-3.5 Turbo. This follows similar moves by Anthropic (Claude 3 Haiku) and Google (Gemini 1.5 Flash).

Key features:

* Significantly smarter than GPT-3.5 Turbo, but not as capable as GPT-4o (see benchmark).
* 60% cheaper compared to GPT-3.5 Turbo.
* It's a vision model, meaning it can interpret images.
* Context window is as big as GPT-4o (128k).

Why a "weaker" model when there is a better one? Quite simply: Not every task needs the best model.

Recommendation: For simple tasks such as summarizing texts or improving formulations, 4o-mini is perfectly sufficient. When more knowledge and complex "thinking" is required, you resort to 4o.

**June 21st, 2024**

## Claude 3.5 Sonnet

Two hours after Claude 3.5 Sonnet was released, it became available on Orq.ai. Toggle it on in the **AI Router** and try it out yourself!

A little about the new model. First off, Claude 3.5 Sonnet can handle both text and images. It's also Anthropic's top-performing model to date, outshining the previous Claude 3 Sonnet and Claude 3 Opus in various AI benchmarks (see image).

One area where Claude 3.5 Sonnet does seem to excel is in speed. Anthropic claims it's about twice as fast as Claude 3 Opus, which could be a game-changer for developers building apps that need quick responses, like customer service chatbots.

The model's vision capabilities have also seen a significant boost. It's better at interpreting charts, graphs, and even reading text from less-than-perfect images. This could open up some interesting possibilities for real-world applications.

However, it's worth noting that Claude 3.5 Sonnet still has the same context window of 200,000 tokens as its predecessor. While this is a decent amount (about 150,000 words), it's not a step up from what we've seen before.

![Claude 3.5 Sonnet benchmark comparison showing performance against Claude 3 Sonnet and Claude 3 Opus.](https://mintcdn.com/orqai/1clR3dRdy0GtV4iZ/images/e79a805-sonnet_3.5.webp?fit=max\&auto=format\&n=1clR3dRdy0GtV4iZ\&q=85\&s=5b145e4524fe40c946c808f55e753fd0)

**June 12th, 2024**

## Prompt library

The new Prompt library lets you create prompt templates that you can reuse throughout the whole platform.

With the filtering capabilities, you can easily find the right prompt for your use case and effectively manage multiple prompts.

Check the new feature out in your workspace or in this interactive walkthrough below.

**June 4th, 2024**

## Llama 3 on Perplexity

Start using the best open source model, hosted on perplexity, in Orq.

You could already use the Llama 3 models hosted on Anyscale and Groq. However, being able to use Llama-3 on Perplexity opens up new possibilities and use cases.

Because the model is able to go online, it is able to:

* Generate up-to-date responses
* Retrieve dynamic data about the latest news etc.
* Understand real-world context better

The example below showcases that only Llama-3 on perplexity is able to generate the current temperature in Amsterdam.

![Comparison showing Llama-3 on Perplexity returning real-time data such as the current temperature in Amsterdam.](https://mintcdn.com/orqai/W8gaV2S7D8ydczdf/images/b82669a-iScreen_Shoter_-_Google_Chrome_-_240604153255.jpg?fit=max\&auto=format\&n=W8gaV2S7D8ydczdf\&q=85\&s=fa7f4f7548c1fb6a680861784bee2dff)

**May 28th, 2024**

## Multi API key selector

You can now select which API key to use in your Playgrounds, Experiments, and Deployments.

Select which API key to use with the new **integration selector** pill.

This only works after you have set up more than one API key. *Note: This feature does not apply on Azure and AWS.*

In the example below, I use the exact same model configuration as my fallback model. However, I will use my second API key in case the first one doesn't work.

<video src="https://mintcdn.com/orqai/1oqtKOJj_dB5Fg2y/videos/integration_pill.mp4?fit=max&auto=format&n=1oqtKOJj_dB5Fg2y&q=85&s=30840942e30dd39c07df373a69de650d" autoPlay muted loop playsInline>
  Animation showing the integration selector pill for switching between multiple API keys in a Deployment.
</video>

**May 27th, 2024**

## Deployment test run

Preview your LLM call in the Deployment studio.

This new feature allows you to quickly do a test run. You still have the option to open the same configuration in the Playground for further testing.

<video src="https://mintcdn.com/orqai/1oqtKOJj_dB5Fg2y/videos/area.mp4?fit=max&auto=format&n=1oqtKOJj_dB5Fg2y&q=85&s=a0e1334285ce7e6b3c30fbd5aaa7c752" autoPlay muted loop playsInline>
  Animation showing the Deployment test run feature for previewing an LLM call directly in the Deployment studio.
</video>

**May 7th, 2024**

## Groq custom integration

Set up your custom API keys with the new Groq integration.

<video src="https://mintcdn.com/orqai/1oqtKOJj_dB5Fg2y/videos/groq_integration.mp4?fit=max&auto=format&n=1oqtKOJj_dB5Fg2y&q=85&s=44eb09620f08f2e30dc416d34c3f010a" autoPlay muted loop playsInline>
  Animation showing how to set up a Groq custom API key integration in Orq.ai.
</video>

**May 7th, 2024**

## Keyboard shortcuts

For the real prompt engineers in Orq, we introduced some keyboard shortcuts in the Playground module. If your cursor is located within the prompt template, you can use the hotkeys below to generate an output.

`cmd/ctrl` + `enter` runs the current block.

`shift` + `cmd/ctrl` + `enter` runs all the blocks.

Feel free to contact us for other shortcut suggestions.

**May 7th, 2024**

## New version pop-up

With the introduction of the new version pop-up, you can now choose when you'd like to do the update. This way, you won't get interrupted in the middle of your workflow.

![New version update pop-up allowing users to choose when to apply a platform update without interrupting their workflow.](https://mintcdn.com/orqai/W8gaV2S7D8ydczdf/images/b34cc31-version_update.jpg?fit=max\&auto=format\&n=W8gaV2S7D8ydczdf\&q=85\&s=def6755ff6a0aa21608e5c92b440ca3e)

**May 7th, 2024**

## Tools workflow

Quickly **import** a **Tool** into a Deployment.

Also, when creating a Tool in a Deployment, you can save it by clicking `Add to resources`.

<video src="https://mintcdn.com/orqai/1oqtKOJj_dB5Fg2y/videos/tools_import.mp4?fit=max&auto=format&n=1oqtKOJj_dB5Fg2y&q=85&s=6fae3083242350f8e32a469ee25ba785" autoPlay muted loop playsInline>
  Animation showing how to import a Tool from the Resources tab into a Deployment.
</video>

Create, save, and find all your **Tools** in the **Resources** tab.

<video src="https://mintcdn.com/orqai/1oqtKOJj_dB5Fg2y/videos/tools_res.mp4?fit=max&auto=format&n=1oqtKOJj_dB5Fg2y&q=85&s=a76d31cd5b45b8e3c4cdec42db29b3ff" autoPlay muted loop playsInline>
  Animation showing how to save and find Tools in the Resources tab.
</video>

**April 26th, 2024**

## New domain for ORQ.ai

> 🚧 If you're using my.orquesta.dev please start using my.orq.ai

**April 26th, 2024**

## Version Controlled Deployments

With this new feature, you can view the changes made to a Deployment and restore to the previous version if needed.

<video src="https://mintcdn.com/orqai/1oqtKOJj_dB5Fg2y/videos/versioncontrol.mp4?fit=max&auto=format&n=1oqtKOJj_dB5Fg2y&q=85&s=a32c08ddefb7da5634cfb7a226a51ed3" autoPlay muted loop playsInline>
  Animation showing the Version Controlled Deployments feature for reviewing changes and restoring previous versions.
</video>

On top of that, when deploying, users get the option to write a quick description about what they changed and if it's a minor or major change.

![Deployment version dialog for adding a description and marking a change as minor or major before deploying.](https://mintcdn.com/orqai/4YNqGRNpuZNyo0_T/images/changelog-deployment-version-dialog.png?fit=max\&auto=format\&n=4YNqGRNpuZNyo0_T\&q=85\&s=cb0c4bd07715e55a0caf1c8e46a3cef0)

**April 24th, 2024**

## Deployment Routing

When opening a Deployment, you now immediately enter the **Prompt Studio**.

To switch to other **Variants**, you can click on the variant name, which opens a dropdown menu with all the other variants that you can choose to switch to. In the same dropdown menu, you can also choose to add another variant.

Under the new **Routing** tab, you can still see the Business Rules Engine and configure it like you're used to.

<video src="https://mintcdn.com/orqai/1oqtKOJj_dB5Fg2y/videos/2switching_between_variants.mp4?fit=max&auto=format&n=1oqtKOJj_dB5Fg2y&q=85&s=fe28b8968a9bd71d4dae40b68c0ea1a3" autoPlay muted loop playsInline>
  Animation showing the Deployment Routing feature for switching between Variants using a dropdown menu.
</video>

**April 24th, 2024**

## Prompt generator

With this new feature, we can help you turn your prompt into a more detailed and complete prompt.

You can also use it to create a template which you can iterate on.

<video src="https://mintcdn.com/orqai/1oqtKOJj_dB5Fg2y/videos/promptgeneration.mp4?fit=max&auto=format&n=1oqtKOJj_dB5Fg2y&q=85&s=72e9fe177cfffed322471ab9591dcd83" autoPlay muted loop playsInline>
  Animation showing the prompt generator turning a basic prompt into a more detailed and complete template.
</video>

**April 23rd, 2024**

## Tools in Experiments

Start using **Tools**, also known as **Function Calling**, in an Experiment.

Once you have set up your Tools in [the Resource Tab](https://docs.orq.ai/ai-studio/ai-engineering/create-tools.md), you can add them in the **configuration** tab in your Experiment.

A strong feature is to use Tools in combination with the Valid JSON and Valid JSON Schema **Evaluators**.

<video src="https://mintcdn.com/orqai/1oqtKOJj_dB5Fg2y/videos/tools_in_experiments.mp4?fit=max&auto=format&n=1oqtKOJj_dB5Fg2y&q=85&s=5cbec8282a8d18f8f4e1a61ce6f6363e" autoPlay muted loop playsInline>
  Animation showing how to add Tools to an Experiment configuration for function calling evaluation.
</video>

**April 23rd, 2024**

## Breadcrumb navigation

With breadcrumbs, you'll always know in which module you're working. On top of that, breadcrumbs allow for quick navigation within the module. See an example below.

![Breadcrumb navigation bar showing the current module and path for quick navigation within Orq.ai.](https://mintcdn.com/orqai/W8gaV2S7D8ydczdf/images/b5eff3a-breadcrumbs.jpg?fit=max\&auto=format\&n=W8gaV2S7D8ydczdf\&q=85\&s=e5abf237a7ed113830c350c5d442cddc)

**April 23rd, 2024**

## Experiment aggregate metrics

With the new Experiment aggregate metrics, you can do a much more accurate comparison between models and/or configurations.

These filters are crucial when doing large Experiments. The heatmap still provides you with a quick glance at what the differences are, but the filter allows you to make a more precise comparison.

The filter includes the sum, average, median, min, max, range, and standard deviation.

<video src="https://mintcdn.com/orqai/1oqtKOJj_dB5Fg2y/videos/filter_experiments_720p30fps.mp4?fit=max&auto=format&n=1oqtKOJj_dB5Fg2y&q=85&s=07a9cbabf663d1da962ef19f00bc1278" autoPlay muted loop playsInline>
  Animation showing the Experiment aggregate metrics filter with sum, average, median, and standard deviation options.
</video>

**April 23rd, 2024**

## Set up your Tools in Resources

Set up your **Tools** in the **Resources** tab.

This update allows you to reuse your Tools across the platform. Also, if you want to use Tools in an Experiment, you have to set them up within the Resources tab first in order to select them.

![The Tools setup screen in the Resources tab for creating and storing reusable function tools.](https://mintcdn.com/orqai/1clR3dRdy0GtV4iZ/images/f5e0323-tools.jpg?fit=max\&auto=format\&n=1clR3dRdy0GtV4iZ\&q=85\&s=7d47356e52bd4de88cb6d4b07da8a1fd)

**April 11th, 2024**

## Google Gemini Pro 1.5

Start using Google's Gemini Pro 1.5 across all three modules on Orq. The newest model from Google is not only more advanced than its predecessor but also boasts several key differentiating features.

Top reasons why you should start testing Gemini Pro 1.5 today:

* 1 million context window! - Allowing it to understand a much greater context than before.
* Complex multimodality reasoning - It's able to understand video, audio, code, image, and text.

  ![Google Gemini Pro 1.5 model with a 1 million token context window available in the AI Router.](https://mintcdn.com/orqai/kH7ELCU0voLJRri0/images/3592c1f-gemini_pro_1.5_x.jpg?fit=max\&auto=format\&n=kH7ELCU0voLJRri0\&q=85\&s=d35f12929de56f00623bb567cc49fc19)

**April 8th, 2024**

## Vision models are now available

Start using vision models to unlock the even greater potential of Generative AI.

Vision models and image models are different. While image models create images, vision models interpret them.

We currently support the following vision models:

* Claude-3-haiku
* Claude-3-sonnet
* Claude-3-opus
* GPT-4-turbo-vision
* GPT-4-vision-preview
* Gemini-pro-vision

  ![Vision model applied to an insurance use case, interpreting an uploaded document or claim image.](https://mintcdn.com/orqai/kH7ELCU0voLJRri0/images/1813de3-insurance_vision.jpg?fit=max\&auto=format\&n=kH7ELCU0voLJRri0\&q=85\&s=992f11d5171d772edb868aa729777b82)

**April 8th, 2024**

## New Cohere models - Command R & Command R+

We now support the latest models from Cohere: Command R & Command R+.

Command R is a highly advanced language model designed to facilitate smooth and natural conversations and perform complex tasks requiring long context. It belongs to the "scalable" category of models that provide both high performance and accuracy.

The most important key strengths of Command R:

* High precision on retrieval augmented generation (RAG) and tool use tasks (function calling)
* Low latency and high throughput
* Large context window 128,000-tokens
* Multilingual capabilities across 10 key languages, with pre-training data for 13 additional languages

  ![Cohere Command R and Command R+ models now available in the Orq.ai AI Router.](https://mintcdn.com/orqai/W8gaV2S7D8ydczdf/images/908d8da-cohere_model_garden.jpg?fit=max\&auto=format\&n=W8gaV2S7D8ydczdf\&q=85\&s=c0f34dbb5336936fb03b8d587be67103)

**April 4th, 2024**

## Prompt Studio V2

Today we released the new Prompt Studio V2. Similar to the updated Playground V2, the Prompt Studio is designed for intuitive prompt engineering and consistency across the entire platform.

The Prompt Studio can be found when clicking on a variant in the business rules engine. Within the Prompt Studio, you can configure your LLM call. It supports function calling (tools), variable inputs, retries, and fallbacks.

Check out the interactive walkthrough below to test it out yourself.

**April 4th, 2024**

## Anthropic Custom Integration

Start using your own API keys with the new Anthropic integration.

<video src="https://mintcdn.com/orqai/1oqtKOJj_dB5Fg2y/videos/anthropic_integration.mp4?fit=max&auto=format&n=1oqtKOJj_dB5Fg2y&q=85&s=138918524e9e0642e542274641930c18" autoPlay muted loop playsInline>
  Animation showing how to set up an Anthropic custom API key integration in Orq.ai.
</video>

**April 4th, 2024**

## AWS Bedrock Custom Integration

Start using your own API keys with the new AWS Bedrock integration.

<video src="https://mintcdn.com/orqai/1oqtKOJj_dB5Fg2y/videos/aws_integration.mp4?fit=max&auto=format&n=1oqtKOJj_dB5Fg2y&q=85&s=ae2faaa59050e962fbabbd37689ea97d" autoPlay muted loop playsInline>
  Animation showing how to set up an AWS Bedrock custom API key integration in Orq.ai.
</video>

**March 16th, 2024**

## New SDKs with breaking changes!

> 🚧 **`We recently transitioned our SDK's from @orquesta to @orq-ai across all platforms to align with our branding and focus on AI. This includes some breaking changes!`**

## **Python SDK**

### Installation

Before

```shell theme={"theme":{"light":"github-light","dark":"github-dark"}}
pip install orquesta-sdk
```

After

```shell theme={"theme":{"light":"github-light","dark":"github-dark"}}
pip install orq-ai-sdk
```

### Usage

Before

```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from orquesta_sdk import Orquesta, OrquestaClientOptions

api_key = os.environ.get("ORQUESTA_API_KEY", "__API_KEY__")

options = OrquestaClientOptions(
    api_key=api_key,
    environment="production"
)

client = Orquesta(options)
```

After

```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os

from orq_ai_sdk import OrqAI

client = OrqAI(
   api_key=os.environ.get("ORQ_API_KEY", "__API_KEY__"),
   environment="production"
)

```

## **Node SDK**

### Installation

Before

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
npm install @orquesta/node
```

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
yarn add @orquesta/node
```

After

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
npm install @orq-ai/node
```

```bash theme={"theme":{"light":"github-light","dark":"github-dark"}}
yarn add @orq-ai/node
```

### Usage

Before

```typescript Typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { createClient } from '@orquesta/node';

const client = createClient({
  apiKey: 'orquesta-api-key',
  environment: 'production',
});
```

After

```typescript Typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { createClient } from '@orq-ai/node';

const client = createClient({
  apiKey: 'orquesta-api-key',
  environment: 'production',
});
```

**March 14th, 2024**

## Playground V2

Today, we launched our new Playground V2. This updated layout is designed for prompt engineering. By keeping the prompt template separate from the chat messages, it's easier to fine-tune your prompt. When you clear the chat messages, the prompt template will still be there.

Check out our new Playground V2 in the interactive walkthrough below!

**March 14th, 2024**

## Claude 3 - Haiku, Sonnet and Opus are now available

With the launch of Claude 3, Anthropic is challenging the status quo. Its most advanced model 'Opus' is supposedly better than GPT-4. However, it's also around 2.5 times more expensive than GPT-4. Haiku, the cheapest and least capable model, is around the same price as GPT-3.5 and is really fast.

On another note, none of the Claude models support image and video generation. However, they do have the ability to interpret and analyze images. Moreover, they do have a default context window of 200k tokens, making it useful for larger data sets and variables.

The Sonnet and Haiku models are available on Anthropic as well as on AWS Bedrock. Whereas Opus is currently only available on Anthropic.

You can toggle them on in the **AI Router** and try them out yourself!

![Claude 3 benchmark comparison showing Haiku, Sonnet, and Opus performance relative to GPT-4 and other models.](https://mintcdn.com/orqai/eNeVsDsn_DMSI7MV/images/changelog-claude-3-benchmark.jpg?fit=max\&auto=format\&n=eNeVsDsn_DMSI7MV\&q=85\&s=d728d70edee936959bcbfbb7bc4310ee)

**March 14th, 2024**

## Hyperlinking

With the new Hyperlinking feature, you are able to take your use case from one module to the other. Switching between the Playground, Experiments, and Deployments allows you to make quick iterations throughout the whole platform. Whether you want to take your Playground setup to Experiments or your Deployment to Playground, it's all possible.

Example: You are running a Deployment but want to quickly test out what would happen if you change your prompt. You don't want to change the prompt within your Deployment because it's in production. With the new Hyperlinking feature, you can open the exact same configuration used in one module in the other. Follow these steps to try it out yourself:

1. Navigate towards the Logs tab
2. Click on the log that has the configuration that you want to iterate on
3. Click on the three-dotted menu in the upper right corner
4. Click open in Playground
5. Click on 'View Playground'
6. You'll be redirected to the Playground and see the exact same configuration you are using in your Deployment

<video src="https://mintcdn.com/orqai/1oqtKOJj_dB5Fg2y/videos/hyperlinking_720.mp4?fit=max&auto=format&n=1oqtKOJj_dB5Fg2y&q=85&s=4023aa5f5509c96fb667a329462de8c1" autoPlay muted loop playsInline>
  The Hyperlinking feature opening a Deployment log configuration directly in the Playground for iteration.
</video>

**March 6th, 2024**

## Use Groq as your LLM provider

You can now use Groq as your LLM provider. It currently hosts the following models: llama2-70b-chat, mixtral-8x7b-32768, and Gemma-7b-it.

**What is Groq and why should I try them out?**

Groq has recently gained a lot of attention for being one of the top companies providing fast LLM inference solutions. There's a growing need among Gen AI application builders to decrease response times, as lowering latency is key for real-time AI applications. The field is getting crowded, with many companies competing in LLM inference. Groq being one of them, claims to offer inference speeds that are 18 times faster than some well-known cloud-based services. The models hosted by Groq utilize the Groq LPU™ Inference Engine, which is powered by their specialized hardware called Language Processing Units (LPUs) to enhance LLM performance.

![Groq LLM inference tokens per second benchmark showing speeds compared to other providers.](https://mintcdn.com/orqai/kH7ELCU0voLJRri0/images/237f5a9-output_tokens_per_s.jpg?fit=max\&auto=format\&n=kH7ELCU0voLJRri0\&q=85\&s=28be96fa4458af964531f85ae82a34f6)

*Tokens per second*

**March 1st, 2024**

## Make your own custom LLM Evaluator

You can now create your own custom LLM evaluator. This allows you to go beyond the standard evaluators like BLEU and Valid JSON.

In the example below, we made an evaluator that compares the tone of voice of the generated text against the desired tone in the reference.

Read more about [Experiments](https://docs.orq.ai/ai-studio/optimize/experiments.md) and [Evaluators](https://docs.orq.ai/ai-studio/optimize/evaluators.md)

![A custom LLM Evaluator comparing the tone of voice of generated text against a desired tone from the reference.](https://mintcdn.com/orqai/kH7ELCU0voLJRri0/images/2b03ae3-iScreen_Shoter_-_Google_Chrome_-_240229173159.jpg?fit=max\&auto=format\&n=kH7ELCU0voLJRri0\&q=85\&s=a284c2d70e3c6d0e2c29d4e60c1d2558)

**February 28th, 2024**

## Use LLM as a Reference in Experiments

With this new feature, you're able to use the output of a large language model like GPT-4 as the reference for another model like Gemma-7b and Mistral-large (see image).

For most of the evaluators, you need a reference. This is because an eval like cosine similarity needs two things to compare to each other (the newly generated text and the reference text).

Example use case: A new model has been released which is faster and less expensive than the model you are currently using. Although your current model is performing well, you are interested in comparing the performance of the new model. To compare the two models, you have selected your current model (GPT-4) as the reference model in the configuration. This will serve as a benchmark for the new model's performance. When running the experiment, the reference model will be completed first, and then the output of that model will be used as a reference for the other models. To measure the similarity between the output of the newer models and the reference model, you can use an evaluator such as cosine similarity.

![An Experiment using GPT-4 as the reference model to benchmark a newer model via cosine similarity evaluation.](https://mintcdn.com/orqai/4YNqGRNpuZNyo0_T/images/changelog-experiment-gpt4-benchmark.png?fit=max\&auto=format\&n=4YNqGRNpuZNyo0_T\&q=85\&s=b6d72d048ba13c5aca8a5babd5cfecf0)

**February 28th, 2024**

## New Providers & Models: Mistral large, Perplexity, Gemma 7b and other models are added

**February 28th, 2024**

## Import from Resources in Experiments

Instead of manually adding your data sets or uploading them through a CSV file in experiments, you can now import them from the files you stored in Resources. This allows you to store and access your files in a quick and efficient manner.

![Importing a dataset from the Resources files tab into an Experiment instead of uploading a CSV manually.](https://mintcdn.com/orqai/4YNqGRNpuZNyo0_T/images/changelog-experiment-dataset-import.png?fit=max\&auto=format\&n=4YNqGRNpuZNyo0_T\&q=85\&s=5a89c10fd0e82b6504a305d1253142c3)

**February 28th, 2024**

## Resource management

Save your data sets, variables, and evaluators in the newly added resources tab.

With this new feature, you don't have to constantly re-upload your resources anymore.

Example: After finetuning your prompt, simply save the prompt in the resources tab so you can easily access it during your next workflow.

The resources are divided into 3 categories:

Data sets - This is where you can store all your prompts and references. Read more about [Dataset](https://docs.orq.ai/ai-studio/optimize/datasets.md).

Variables - The dynamic elements you put in between curly\_brackets are listed here.

Evaluators - The standard and custom evals can be stored here. Read more about why you should use evals and when to use which one in [Evaluator](https://docs.orq.ai/ai-studio/optimize/evaluators.md).

![The Resources tab showing datasets, variables, and evaluators organized for reuse across the platform.](https://mintcdn.com/orqai/4YNqGRNpuZNyo0_T/images/changelog-resources-tab.jpg?fit=max\&auto=format\&n=4YNqGRNpuZNyo0_T\&q=85\&s=2073f21b3f134ba0cb3384ef52686b9c)

**February 13th, 2024**

## Evaluators are available on our Platform

We have added Evaluators to our platform. With a wide range of industry-standard metrics and other relevant evaluators, you can check whether or not the output of your LLM is accurate, reliable, and contextually relevant.

Read more about each evaluator in [Evaluators](https://docs.orq.ai/ai-studio/optimize/evaluators.md).

<video src="https://mintcdn.com/orqai/1oqtKOJj_dB5Fg2y/videos/evals720p4.mp4?fit=max&auto=format&n=1oqtKOJj_dB5Fg2y&q=85&s=553899474e41b60043999eb82d82b28d" autoPlay muted loop playsInline>
  The Evaluators launch showing industry-standard metrics available for checking LLM output accuracy and relevance.
</video>

**February 13th, 2024**

## Python and Node SDK's improvements

In our latest SDK update, we're thrilled to share a series of enhancements that significantly boost the performance and capabilities of our platform.

## Support for messages in the invoke and stream functionalities

Now you can provide a new property in the `invoke` and the `invoke with stream` methods of both SDKs to combine the messages with the prompt configured in Orquesta.

```typescript Typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const deployment = await client.deployments.invoke({
  key: 'customer_service',
  messages: [
    {
      role: 'user',
      message:
        'A customer is asking about the latest software update features. Generate a detailed and informative response highlighting the key new features and improvements in the latest update.',
    },
  ],
  context: { environments: 'production', country: 'NLD' },
  inputs: { firstname: 'John', city: 'New York' },
  metadata: { customer_id: 'Qwtqwty90281' },
});

console.log(deployment?.choices[0].message.content);
```

The introduction of the new property significantly enhances various aspects of interaction:

1. **Enhanced Contextual Clarity**: The use of chat history empowers the model to preserve the context throughout a conversation. This feature ensures that each response is not only coherent but also directly relevant, as the model can draw upon past dialogues to fully grasp the nuances of the current inquiry or discussion topic.
2. **Streamlined Conversation Flow**: Chat history is instrumental in maintaining a consistent and logical flow in conversations. This prevents the occurrence of repetitive or conflicting responses, mirroring the natural progression of human dialogues and maintaining conversational integrity.
3. **Tailored User Interactions**: With access to previous interactions, chat history allows the model to customize its responses according to individual user preferences and historical queries. This level of personalization significantly boosts user engagement and satisfaction, leading to more effective and enjoyable communication experiences.

## Support for messages and choices to add metrics

In Orquesta, after every request, you can add custom metrics. On top of the superset that we already support, we also added two new properties, `messages` and `choices`. The `messages` property is designed to log the communication exchange between the user and the model, capturing the entire conversation history for context and analysis. Meanwhile, the `choices` property records the different response options that the model considers or generates before presenting the final output, providing insights into the model's decision-making process.

```typescript Typescript theme={"theme":{"light":"github-light","dark":"github-dark"}}
deployment.addMetrics({
  messages: [
    {
      role: 'user',
      message:
        'A customer is asking about the latest software update features. Generate a detailed and informative response highlighting the key new features and improvements in the latest update.',
    },
  ],
  choices: [
    {
      index: 0,
      finish_reason: 'stop',
      message: {
        role: 'assistant',
        content:
          "Dear customer: Thank you for your interest in our latest software update! We're excited to share with you the new features and improvements we've rolled out. Here's what you can look forward to in this update",
      },
    },
  ],
});
```

## Support for OpenAI system fingerprint

To help track OpenAI environmental or model changes, OpenAI is exposing a `system_fingerprint` parameter and value. If this value changes, different outputs will be generated due to changes to the OpenAI environment.

In the new version of the SDK, if you are using an OpenAI model and the `invoke` method, the `system_fingerprint` will be exposed in the deployment properties.

**January 25th, 2024**

## Setup your own API key with the new AnyScale integration.

You could already select models such as Llama from Meta and Mixtral from Mistral in the **AI Router**. But with this release, it is now possible to connect your own API key for AnyScale. This way you can use your own account and rate limits without having to rely on a shared key. Soon you'll be able to use your own private models and finetuning on AnyScale.

<video src="https://mintcdn.com/orqai/1oqtKOJj_dB5Fg2y/videos/anyscale_api.mp4?fit=max&auto=format&n=1oqtKOJj_dB5Fg2y&q=85&s=51baa16013ee480d44f9d035f5f727ac" autoPlay muted loop playsInline>
  Setting up a custom AnyScale API key integration to use personal rate limits with Llama and Mixtral models.
</video>

**January 25th, 2024**

## Mass Experimentation

You could already test out different models and configurations using our playground. However, with the introduction of Experiments, you are able to do this on a much larger scale.

Simply import your prompts, expected outputs, and variables, configure your model, and you're able to do hundreds of simultaneous runs. This allows you to do a side-by-side comparison in large batches.

Check out the interactive walkthrough.

**January 3rd, 2024**

## Python and Node SDK's now support multimodality

We improved our SDKs to support more types of models. Now, it's possible to use completion, chat, and image models through our unified API.

We also improved our API reference [here](https://docs.orq.ai/reference/deployments/invoke.md) to offer a better DX.

**Python SDK**

[https://pypi.org/project/orquesta-sdk/](https://pypi.org/project/orquesta-sdk/)

**Node SDK**

[https://www.npmjs.com/package/@orq-ai/node](https://www.npmjs.com/package/@orq-ai/node)

**January 3rd, 2024**

## Function calling supported in the Playground

For a better UX on Orquesta, we improve how to handle function calling in the Playground and the Prompt Studio.

Now you can simply enter your function name, description, and parameters with the JSON schema format.

![Function calling configuration in the Playground with function name, description, and JSON schema parameters.](https://mintcdn.com/orqai/W8gaV2S7D8ydczdf/images/8f607e5-image.png?fit=max\&auto=format\&n=W8gaV2S7D8ydczdf\&q=85\&s=b993c047b446a5d18d02b76c7922c1d9)

**January 3rd, 2024**

## Billable transactions are now shown in the log details

Now, it is possible to see in the log details if the request to the LLM was executed with Orquesta API keys or your API keys.

**January 3rd, 2024**

## Gemini, Google’s most capable model, now available on Orquesta

Now, using Gemini Pro on Orquesta through the Playground and Deployments is possible. This model also supports function calling.

**January 1st, 2024**

## Log and metrics enhancement

We introduced a new log detail drawer for a better and more segmented data structure. This will allow our users to consume the information more easily.

**January 1st, 2024**

## Chat history now available in the playground

If you are using the Playground, you can now utilize **Playground memory**. This feature enables you to add historical context when comparing different models simultaneously.

For **completion models**, we strive to incorporate historical context, but this may occasionally lead to undesired effects in the final completions. If you encounter such issues, we recommend disabling **Playground memory**.

**January 1st, 2024**

## Support for Dall-E models

Added support for Dall-E models in the Playground and Deployments for OpenAI and Azure.

![Dall-E model support in the Playground and Deployments for OpenAI and Azure.](https://mintcdn.com/orqai/W8gaV2S7D8ydczdf/images/7fb6822-image.png?fit=max\&auto=format\&n=W8gaV2S7D8ydczdf\&q=85\&s=cf8adf91495b2dbfb46465f6769caf5e)

**January 1st, 2024**

## Orq.ai Integrations

We added a new Integrations section to allow the use of your API keys. At the moment, we support custom API keys from the following LLM providers

* OpenAI
* Azure
* Cohere

More providers and integrations are coming soon.

![The Integrations page showing support for custom API keys from OpenAI, Azure, and Cohere.](https://mintcdn.com/orqai/kH7ELCU0voLJRri0/images/6b9f825-image.png?fit=max\&auto=format\&n=kH7ELCU0voLJRri0\&q=85\&s=a009538638660eafe3263ce87f863833)

**January 1st, 2024**

## Reset models default parameter in a Deployment

Now, it is possible to reset the parameters of the primary model and the fallback model in a Deployment.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.