> ## Documentation Index > Fetch the complete documentation index at: https://docs.orq.ai/llms.txt > Use this file to discover all available pages before exploring further. # Deployments > Ship LLM use cases to production with Orq.ai. Configure model routing, invoke via API or SDK, and monitor in real time. **Deployments** ship Gen AI use cases to production with **Orq.ai** as an AI Gateway. All calls route through the platform, providing routing, monitoring, and security in one place. Connect with a single line of code, iterate without a code release, and benefit from full observability throughout. Common use cases include customer support bots, RAG-powered document Q\&A, content generation pipelines, and any LLM feature that needs reliable model routing, versioning, and production monitoring. Set up a Deployment with a key, model, and system prompt in AI Studio or via MCP. Set the model, fallbacks, variables, knowledge base, tools, caching, and guardrails per Variant. Route traffic across Variants by environment, context attributes, or percentage split. Deploy and roll back configurations without a code release. Call a Deployment via API or SDK and pass identity, usage tracking, and extra parameters. Monitor requests, filter logs by Variant, and inspect full request details. ## Create a Deployment Choose a [Project](/docs/projects/overview) and folder, then select the `+` button. Select **Deployment** from the entity picker. Create Deployment dialog with fields for Deployment Key set to key123 and Model set to claude-3-7-sonnet-20250219.

Create Deployment dialog with fields for Deployment Key set to key123 and Model set to claude-3-7-sonnet-20250219.

Set the deployment key (alphanumeric) and select the primary model for the first Variant. The Variant editor opens. Use the [Orq MCP server](/docs/integrations/code-assistants/mcp) to manage deployments directly from an AI code assistant. **Find an existing deployment:** ```prompt wrap theme={"theme":{"light":"github-light","dark":"github-dark"}} Search for the "support-bot" deployment in my workspace ``` The assistant uses `search_entities` with `type: "deployment"` to locate deployments by name or key. *** **Retrieve deployment configuration:** ```prompt wrap theme={"theme":{"light":"github-light","dark":"github-dark"}} Get the full configuration of the "support-bot" deployment ``` The assistant uses `get_deployment` to return the key, description, model, messages, and variant settings. *** **Create a deployment:** ```prompt wrap theme={"theme":{"light":"github-light","dark":"github-dark"}} Create a customer support deployment called "support-bot" in the Default/Deployments project. Use GPT-4o with a professional, concise system prompt. ``` The assistant uses `create_deployment` with the specified `key`, `path`, and `variant` (model and messages). Use `list_models` first to find valid model IDs. ## Configure a Variant **Variants** are different prompt and model configurations available behind one Deployment. A Deployment can hold any number of Variants. On creation, the **Variant** screen opens for model and prompt setup. A Variant Prompt is similar to any other prompt. To learn how to configure a Prompt, see [Creating a Prompt](/docs/prompts/creating). ### Primary Model, Retries, and Fallback The **Primary Model** panel defines the first model queried through this Variant. **Retries** In case of failure, configure how many times a query is retried with this model. Retries are only triggered when a retry count greater than 0 is configured in the Variant settings. When retries are enabled, **Orq.ai** automatically retries the model provider API call if it returns one of the following HTTP status codes: * 429 Rate Limit Exceeded * 500 Internal Server Error * 501 Not Implemented * 502 Bad Gateway * 503 Service Unavailable **Error handling flow:** 1. If an error code above is returned and retries are configured (retry count > 0), **Orq.ai** retries the Primary Model. 2. If all retry attempts fail (or no retries are configured) AND a Fallback Model is configured, **Orq.ai** routes to the Fallback Model. 3. If the Fallback Model also fails, the error is returned to the calling application. **Fallback Model** The Fallback Model is triggered only if the Primary Model fails after all configured retries are exhausted. Fallback Models can have a different configuration from the Primary Model. Primary Model section showing claude-opus-4-20250514 with Fallback Models configured to gpt-5.2 with reasoning effort, verbosity, and response format settings.

Primary Model section showing claude-opus-4-20250514 with Fallback Models configured to gpt-5.2 with reasoning effort, verbosity, and response format settings.

Multiple fallback models can be configured in a Deployment. They fall back to one another in order of configuration. Use the **Add extra fallback** button to declare another model. See how fallbacks and retries work together in a production system. Read our cookbook [Customer Support Chat](/docs/tutorials/buildingcustomersupportchatwithaigateway). **API invocation behavior** When invoking a Deployment via the API, response timing depends on the retry and fallback configuration: * **Success on first try**: Response returned immediately. * **Retry scenario**: Response may be delayed by up to `base_latency × (retry_count + 1)` to account for the initial attempt plus all configured retries. * **Fallback invoked**: Additional latency as the Fallback Model processes the request. * **All retries and fallback failed**: Error returned to the calling application. Set appropriate timeouts on API calls to account for retry and fallback latency. ### Structured Outputs Configure **structured outputs** to ensure consistent and reliable responses from a Deployment. Structured outputs specify the exact format the model should follow when generating a response. Two modes are available: * **JSON Mode**: the model automatically returns a valid JSON object for every generation. * **JSON Schema**: define a schema that explicitly describes the fields, types, and structure of the model output. Once defined, a schema can be saved to the directory for reuse across multiple variants or deployments. Primary Model settings with Response Format set to JSON Schema, showing a schema selector dropdown with get_weather and json_p3ft options.

Primary Model settings with Response Format set to JSON Schema, showing a schema selector dropdown with get_weather and json_p3ft options.

### Variables and Prompt Templating Reference dynamic values in the prompt using double braces: `{{variable_name}}`. Pass a key-value map to the `inputs` field when invoking and **Orq.ai** substitutes each variable before sending the prompt to the model. **Orq.ai** supports three template engines. Select the **Template Engine** from the Variant Settings panel: * **Text** (default): variables use `{{double_braces}}` syntax. * **Jinja**: full templating with conditionals, loops, filters, and more. * **Mustache**: logic-less templating with sections. Template Engine dropdown with Text currently selected and options for Jinja and Mustache.

Template Engine dropdown with Text currently selected and options for Jinja and Mustache.

**Example: support bot that adapts by subscription tier** ```jinja Jinja theme={"theme":{"light":"github-light","dark":"github-dark"}} You are a support assistant for {{company_name}}. {% if user_tier == "premium" %} {{customer_name}} is a premium customer. Greet them by name and let them know they have priority support with a 2-hour response SLA. {% else %} {{customer_name}} is on the free plan. Let them know the standard response time is 24 hours. {% endif %} ``` System prompt in the Studio editor showing a Jinja template with if/else blocks for premium and free tier customers using is_premium, customer_name, and company_name variables.

System prompt in the Studio editor showing a Jinja template with if/else blocks for premium and free tier customers using is_premium, customer_name, and company_name variables.

```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}} response = client.deployments.invoke( key="support-bot", inputs={ "company_name": "Acme", "customer_name": "Sarah", "user_tier": "premium", } ) ``` ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}} const response = await client.deployments.invoke({ key: "support-bot", inputs: { company_name: "Acme", customer_name: "Sarah", user_tier: "premium", }, }); ``` Trace view showing a rendered Jinja template for gpt-3.5-turbo with company_name set to Acme, customer_name to Sarah, and is_premium to true, generating a priority support greeting.

Trace view showing a rendered Jinja template for gpt-3.5-turbo with company_name set to Acme, customer_name to Sarah, and is_premium to true, generating a priority support greeting.

```handlebars Mustache theme={"theme":{"light":"github-light","dark":"github-dark"}} You are a support assistant for {{company_name}}. {{! Pass is_premium: true for premium customers, false for free plan }} {{# is_premium}} {{customer_name}} is a premium customer. Greet them by name with priority support and a 2-hour SLA. {{/ is_premium}} {{^ is_premium}} {{customer_name}} is on the free plan. Standard response time is 24 hours. {{/ is_premium}} ``` $System prompt in the Studio editor showing a Mustache template with {{#is_premium}} and {{^is_premium}} sections for premium and free plan customers.$ ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}} response = client.deployments.invoke( key="support-bot", inputs={ "company_name": "Acme", "customer_name": "Sarah", "is_premium": True, } ) ``` ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}} const response = await client.deployments.invoke({ key: "support-bot", inputs: { company_name: "Acme", customer_name: "Sarah", is_premium: true, }, }); ``` Trace view showing a rendered Mustache template for gpt-3.5-turbo with company_name set to Acme, customer_name to Sarah, and is_premium to true, with the assistant greeting Sarah as a premium customer.

Trace view showing a rendered Mustache template for gpt-3.5-turbo with company_name set to Acme, customer_name to Sarah, and is_premium to true, with the assistant greeting Sarah as a premium customer.

Add `{{variable_name}}` placeholders to the prompt and pass the corresponding values in the `inputs` field at invoke time. **Orq.ai** substitutes each key before sending the prompt to the model. ```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}} curl --request POST \ --url https://api.orq.ai/v2/deployments/invoke \ --header 'Authorization: Bearer ' \ --header 'Content-Type: application/json' \ --data '{ "key": "my-deployment", "context": {"environments": "production"}, "inputs": { "customer_name": "John Smith", "user_tier": "premium" } }' ``` ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}} generation = client.deployments.invoke( key="my-deployment", context={"environments": "production"}, inputs={ "customer_name": "John Smith", "user_tier": "premium", }, ) print(generation.choices[0].message.content) ``` ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}} const generation = await client.deployments.invoke({ key: 'my-deployment', context: { environments: 'production' }, inputs: { customer_name: 'John Smith', user_tier: 'premium', }, }); console.log(generation.choices[0].message.content); ``` For a complete reference of all template features including filters, macros, nested objects, and more, see [Prompt Templating](/docs/prompts/templating). To prevent sensitive input values from appearing in traces and logs, see [Security and Privacy](/docs/deployments/creating#security-and-privacy). ### Knowledge Base Ground a Deployment's responses in domain-specific knowledge by adding a [Knowledge Base](/docs/knowledge/overview). Open the deployment configuration, go to **Knowledge Bases**, then select **Knowledge Base**. Knowledge Bases enable RAG (Retrieval-Augmented Generation), allowing the model to retrieve and use relevant information from documentation or data sources to provide more accurate and contextual responses. **Configuration options** (via the `...` menu on an attached Knowledge Base): * **Last User Message**: the user's latest message is automatically used as a query to retrieve relevant chunks. * **Query**: a predefined query is used to retrieve chunks. Use Input Variables like `{{query}}` to make it dynamic at runtime. Edit Knowledge Base dialog with Knowledge Base set to knowledge and Type set to Last User Message.

Edit Knowledge Base dialog with Knowledge Base set to knowledge and Type set to Last User Message.

To learn more about creating and configuring Knowledge Bases, see [Knowledge Bases](/docs/knowledge/overview). Reference the Knowledge Base in the prompt using the `{{knowledge_base_key}}` syntax, where `knowledge_base_key` is the identifier of the Knowledge Base. If the Knowledge Base is not explicitly referenced in the prompt, retrieved chunks are automatically appended to the end of the system message. Deployment settings showing a Knowledge Base named knowledge in the settings panel, with the {knowledge} variable highlighted in the system prompt.

Deployment settings showing a Knowledge Base named knowledge in the settings panel, with the {knowledge} variable highlighted in the system prompt.

See knowledge base retrieval used end-to-end in a working deployment. Read our cookbook [Multilingual FAQ Bot](/docs/tutorials/multilingual-faq-bot). When invoking a Deployment that uses a Knowledge Base, set `include_retrievals: true` in `invoke_options` to embed the retrieval chunks in the response. ```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}} curl --location 'https://api.orq.ai/v2/deployments/invoke' \ --header 'Content-Type: application/json' \ --header 'Accept: application/json' \ --header 'Authorization: Bearer xxxxx' \ --data '{ "key": "deployment_key", "messages": [ { "role": "user", "content": "" } ], "invoke_options": { "include_retrievals": true } }' ``` ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}} generation = client.deployments.invoke( key="deployment_key", messages=[ { "role": "user", "content": "" } ], invoke_options={"include_retrievals": True} ) print(generation.choices[0].message.content) ``` ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}} const deployment = await client.deployments.invoke({ key: "deployment_key", messages: [ { "role": "user", "content": "" } ], invokeOptions: { includeRetrievals: true }, }); ``` Retrievals are returned in the `retrievals` field of the response. Each chunk includes source details and scores: API response in a REST client showing the retrievals array with document metadata including file names, file type, page number, and search score for apple_annual_report_2023.pdf.

API response in a REST client showing the retrievals array with document metadata including file names, file type, page number, and search score for apple_annual_report_2023.pdf.

```json theme={"theme":{"light":"github-light","dark":"github-dark"}} { "retrievals": [ { "document": "", "metadata": { "file_name": "", "file_type": "application/pdf", "page_number": 24, "search_score": 0.7886787056922913, "rerank_score": 0.19868536 } } ] } ``` See knowledge base retrievals wired into a complete application. Read our cookbook [Multilingual FAQ Bot](/docs/tutorials/multilingual-faq-bot). ### Tools Tools can only be added and configured at the **deployment** level. Only **Function tools** are supported in Deployments, enabling the model to call external functions during execution. To add a Function tool, open the **Tools** tab in the deployment configuration and click **Tool**: * **Create a new Tool**: define a custom function directly within the deployment. * **Import an existing Tool**: select a previously created Function tool from the resource library. Tools section with a CurrentDate tool listed and an Add Tool button.

Tools section with a CurrentDate tool listed and an Add Tool button.

To learn more about creating Function tools, see [Creating Tools](/docs/tools/overview). ### Cache Variant generation can be cached to reduce processing time and cost. When an input is received that matches a cached entry within the Variant, the stored response is returned directly without triggering a new generation. To enable caching, open the **Variant Settings** tab and select **Enabled** in the Caching section. The cache can be manually invalidated at any time by clicking the configuration icon. Cache settings with the Enabled toggle on and an Expires in dropdown open, showing options from 1 hour to 2 weeks.

Cache settings with the Enabled toggle on and an Expires in dropdown open, showing options from 1 hour to 2 weeks.

**TTL (time to live)** corresponds to the amount of time a cached response is stored before being invalidated. Once invalidated, a new LLM generation is triggered. Configure the TTL from the drop-down once Caching is enabled. The cache only works when there is an exact match. Image models are not supported. ### Evaluators and Guardrails [Evaluators](/docs/evaluators/build) and Guardrails are configured as separate sections in the variant settings. Both operate on the generation pipeline but with different behaviours. Flow diagram showing a user query passing through Input Guardrails synchronously, then Deployment Model Generation, then Output Guardrails, with Input and Output Evaluators running asynchronously and fail paths returning an Error Response.

Flow diagram showing a user query passing through Input Guardrails synchronously, then Deployment Model Generation, then Output Guardrails, with Input and Output Evaluators running asynchronously and fail paths returning an Error Response.

**Evaluators** Click **Evaluator** to add an evaluator from the Library. Configure each evaluator as: * **Input evaluator**: runs evaluation on the input sent to the model. * **Output evaluator**: runs evaluation on the output generated by the model. Evaluators run **asynchronously** and never block the response. Guardrails section listing input_contains_pii and output_toxicity, and Evaluators section showing HTTP Evaluator at 15%, with a Sample Rate popover displaying 15%.

Guardrails section listing input_contains_pii and output_toxicity, and Evaluators section showing HTTP Evaluator at 15%, with a Sample Rate popover displaying 15%.

**Guardrails** Click **Guardrail** to add a guardrail-capable evaluator from the Library. A Guardrail runs **synchronously** and will **deny** the generation if its evaluation fails, returning an error to the user. Guardrails can be configured as: * **Input Guardrail**: runs **before** the input is sent to the model. * **Output Guardrail**: runs **after** generation, before client response. **Guardrail behavior when a guardrail fails:** | Behavior | Description | | ------------ | ------------------------------------------------------------------------------------------------------------------- | | **Retry** | Triggers a new generation attempt. Use this when a transient or non-deterministic failure may resolve on retry. | | **Fallback** | Executes the fallback model configured on the Deployment. Use this for a safe default response instead of retrying. | Guardrail behavior is configured per Deployment and applies to all guardrails attached to it. **Output Guardrails and Streaming**: When a deployment is invoked with streaming enabled, output guardrails will be deactivated as they cannot be run effectively on chunks only. See guardrails put to the test against adversarial inputs. Read our cookbook [Red Teaming](/docs/tutorials/red-teaming). ### Security and Privacy **Input Masking** Inputs in a Variant can be flagged as PII (Personally Identifiable Information). This is recommended when processing sensitive user data such as names, email addresses, or phone numbers. To configure this, open the **Security** tab when editing an input and choose **Personally Identifiable Information (PII)** from the Privacy drop-down. Variables section with a Question variable and a privacy dropdown showing None and Personal Identifiable Information (PII) options.

Variables section with a Question variable and a privacy dropdown showing None and Personal Identifiable Information (PII) options.

Flagging an input as PII removes its values from logs and traces. When opening a log or trace, the input is shown in red to indicate it was not logged. The API response itself still includes the PII value. Trace detail for gpt-4o showing a user message say hello to {name} and the assistant reply Hello, [name]! How are you today?

Trace detail for gpt-4o showing a user message say hello to {name} and the assistant reply Hello, [name]! How are you today?

The API response will include the PII, but input and output logs and traces will not be logged in **Orq.ai**. **Output Masking** Enable output masking to hide generated outputs from logs and traces. Head to the **Security tab** in the Variant and enable the **Output masking** toggle. Variables section with city and date variables, and a Masking section with the Output Masking toggle enabled.

Variables section with city and date variables, and a Masking section with the Output Masking toggle enabled.

When Output Masking is enabled, logs and traces will not store the generated response. A masked output field with a striped pattern and a tooltip reading The response from the model was masked due to your deployment settings.

A masked output field with a striped pattern and a tooltip reading The response from the model was masked due to your deployment settings.

## Add a Variant A single Deployment can hold multiple Variants. Multiple Variants can handle different use cases and scenarios within one Deployment, and can be served simultaneously through Routing. To add a new Variant, select the Variant name at the top-left of the screen and choose **Add variant**. Variant context menu with options including Edit, Duplicate, Share, Create Variant, Change, and Delete.

Variant context menu with options including Edit, Duplicate, Share, Create Variant, Change, and Delete.

## Routing Once a Variant is ready to be deployed, configure the routing variables to control which Variant is reached. Open the **Routing** page by selecting **Routing** at the top-left of the panel.