> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orq.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Load balancing across providers

> Distribute LLM requests across providers using latency-based, weight-based, or round-robin routing to optimize costs, run A/B tests, and ensure redundancy.

**Use Cases**

* Distributing traffic across multiple provider accounts to stay within per-key rate limits.
* A/B testing providers by routing a configurable percentage of traffic to each.
* Spreading traffic across multiple providers to reduce blast radius from a single provider outage.
* Maximizing throughput when one provider's capacity is a bottleneck.
* Minimizing response time by routing to the fastest available model.

***

## Quick Start

Distribute requests across multiple providers using weighted routing.

<CodeGroup>
  ```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
  curl -X POST https://my.orq.ai/v3/router/responses \
    -H "Authorization: Bearer $ORQ_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "openai/gpt-5.4-mini",
      "input": "Write a marketing slogan",
      "load_balancer": {
        "type": "weight_based",
        "models": [
          {"model": "openai/gpt-5.4-mini", "weight": 0.7},
          {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.3}
        ]
      }
    }'
  ```

  ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
  import OpenAI from "openai";

  const client = new OpenAI({
    apiKey: process.env.ORQ_API_KEY,
    baseURL: "https://my.orq.ai/v3/router",
  });

  const response = await client.responses.create({
    model: "openai/gpt-5.4-mini",
    input: "Write a marketing slogan",
    load_balancer: {
      type: "weight_based",
      models: [
        { model: "openai/gpt-5.4-mini", weight: 0.7 },
        { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.3 },
      ],
    },
  });

  console.log(response.output_text);
  ```

  ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
  from openai import OpenAI
  import os

  client = OpenAI(
      api_key=os.environ.get("ORQ_API_KEY"),
      base_url="https://my.orq.ai/v3/router",
  )

  response = client.responses.create(
      model="openai/gpt-5.4-mini",
      input="Write a marketing slogan",
      extra_body={
          "load_balancer": {
              "type": "weight_based",
              "models": [
                  {"model": "openai/gpt-5.4-mini", "weight": 0.7},
                  {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.3},
              ],
          }
      },
  )

  print(response.output_text)
  ```

  ```typescript TypeScript (Chat Completions) theme={"theme":{"light":"github-light","dark":"github-dark"}}
  import OpenAI from "openai";

  const client = new OpenAI({
    apiKey: process.env.ORQ_API_KEY,
    baseURL: "https://my.orq.ai/v3/router",
  });

  const response = await client.chat.completions.create({
    model: "openai/gpt-5.4-mini",
    messages: [{ role: "user", content: "Write a marketing slogan" }],
    load_balancer: {
      type: "weight_based",
      models: [
        { model: "openai/gpt-5.4-mini", weight: 0.7 },
        { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.3 },
      ],
    },
  });
  ```
</CodeGroup>

## Configuration

| Parameter              | Type   | Required | Description                                                       |
| ---------------------- | ------ | -------- | ----------------------------------------------------------------- |
| `load_balancer`        | Object | Yes      | Load balancer configuration (top-level)                           |
| `load_balancer.type`   | string | Yes      | Strategy type (`weight_based`, `round_robin`, or `latency_based`) |
| `load_balancer.models` | Array  | Yes      | List of models with weights                                       |
| `models[].model`       | string | Yes      | Model identifier                                                  |
| `models[].weight`      | number | Yes      | Weight assigned to this model (0.001 - 1.0)                       |

**Weight Calculation:**

* Weights are normalized: `[0.4, 0.8]` → `[33%, 67%]`.
* Higher weight = more traffic.
* Minimum weight: `0.001`.
* Weight is ignored for round robin selection; every model in the list still receives an equal share of traffic.

<Warning>
  A matching [Routing Rule](/ai-gateway/configuration/routing-rules#providers-and-traffic-weight) with its own target models configured overwrites this `load_balancer` parameter entirely, regardless of what the request sends. The Routing Rule's own strategy and models are used instead; the two configurations are never merged.
</Warning>

<Note>
  `load_balancer` selects one model per request. It does not retry a failed call against another model in the pool. The top-level `model` is used as-is only when `load_balancer` cannot produce a selection, for example an empty `models` list. To fail over when a call errors, pair `load_balancer` with [Fallbacks](/ai-gateway/features/retries#fallbacks).
</Note>

## Latency-based routing

Route each request to the model with the lowest recently observed latency, instead of a fixed traffic split.

<CodeGroup>
  ```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
  curl -X POST https://my.orq.ai/v3/router/responses \
    -H "Authorization: Bearer $ORQ_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "openai/gpt-5.4-mini",
      "input": "Write a marketing slogan",
      "load_balancer": {
        "type": "latency_based",
        "models": [
          {"model": "openai/gpt-5.4-mini", "weight": 0.5},
          {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.5}
        ]
      }
    }'
  ```
</CodeGroup>

**How selection works:**

* Latency is tracked per model as a running average that weights recent calls more heavily, so the selection adapts quickly when a provider speeds up or slows down.
* With fresh data (samples from the last 5 minutes) for every configured model, the lowest-latency model is selected. A model with no data, or stale data, is probed again instead of being written off.
* Ten percent of selections explore the full pool by configured weight to keep latency measurements fresh. Failed calls are penalized so a fast failure does not look like a fast success.
* Configured weights decide exploration and near-ties: with the two fastest models within 0.5 ms of each other, the higher weight wins. With clear data, the lowest latency always wins.

<Accordion title="Full selection flow">
  ```mermaid theme={"theme":{"light":"github-light","dark":"github-dark"}}
  flowchart TD
      A["New request"] --> B{"10% exploration draw?"}
      B -- "Yes" --> C["Pick a model by configured weight"]
      B -- "No" --> D{"Fresh data for every model? (< 5 min old)"}

      D -- "No" --> E["Probe an unknown or stale model by weight"]
      D -- "Yes" --> F{"Lowest two latencies tie within 0.5 ms?"}

      F -- "Yes" --> G["Higher configured weight wins"]
      F -- "No" --> H["Lowest latency wins"]

      C --> I["Call the model"]
      E --> I
      G --> I
      H --> I

      I --> J["Record duration, or a penalty on failure, into the latency history"]
  ```
</Accordion>

<Card title="See also: Organization-level load balancing" icon="sliders" href="/ai-gateway/configuration/routing-rules#providers-and-traffic-weight" horizontal>
  To apply load balancing across your organization without changing request code, use **Routing Rules** to configure Fallback, Latency, Weighted, and Round Robin strategies at the workspace level.
</Card>

## Weight-based routing

Split traffic across models by percentage weights instead of latency or a fixed rotation.

### Patterns

<CodeGroup>
  ```typescript Weight-based config patterns theme={"theme":{"light":"github-light","dark":"github-dark"}}
  // Equal distribution
  load_balancer: {
    type: "weight_based",
    models: [
      { model: "openai/gpt-5.6-sol", weight: 1.0 },
      { model: "anthropic/claude-sonnet-5", weight: 1.0 },
    ],
  }

  // Cost optimization (cheap model primary)
  load_balancer: {
    type: "weight_based",
    models: [
      { model: "openai/gpt-5.4-mini", weight: 0.8 },
      { model: "openai/gpt-5.6-sol", weight: 0.2 },
    ],
  }

  // A/B testing
  load_balancer: {
    type: "weight_based",
    models: [
      { model: "current-model", weight: 0.9 },
      { model: "experimental-model", weight: 0.1 },
    ],
  }

  // Multi-provider redundancy
  load_balancer: {
    type: "weight_based",
    models: [
      { model: "openai/gpt-5.6-sol", weight: 0.5 },
      { model: "anthropic/claude-sonnet-5", weight: 0.3 },
      { model: "azure/gpt-5.6-sol", weight: 0.2 },
    ],
  }
  ```
</CodeGroup>

### Use cases

| Scenario                | Approach                   | Example                           |
| ----------------------- | -------------------------- | --------------------------------- |
| **Cost optimization**   | Heavy on cheaper models    | 80% GPT-5.4 Mini, 20% GPT-5.6 Sol |
| **Performance testing** | Small traffic to new model | 95% current, 5% experimental      |
| **Provider redundancy** | Split across providers     | 60% OpenAI, 40% Anthropic         |
| **Capacity management** | Distribute during peaks    | Even split across models          |

## Round Robin routing

Rotate through the configured models evenly, one request at a time, instead of splitting traffic by weight or latency.

<CodeGroup>
  ```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
  curl -X POST https://my.orq.ai/v3/router/responses \
    -H "Authorization: Bearer $ORQ_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "openai/gpt-5.4-mini",
      "input": "Write a marketing slogan",
      "load_balancer": {
        "type": "round_robin",
        "models": [
          {"model": "openai/gpt-5.4-mini", "weight": 0.5},
          {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.5}
        ]
      }
    }'
  ```

  ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
  import OpenAI from "openai";

  const client = new OpenAI({
    apiKey: process.env.ORQ_API_KEY,
    baseURL: "https://my.orq.ai/v3/router",
  });

  const response = await client.responses.create({
    model: "openai/gpt-5.4-mini",
    input: "Write a marketing slogan",
    load_balancer: {
      type: "round_robin",
      models: [
        { model: "openai/gpt-5.4-mini", weight: 0.5 },
        { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.5 },
      ],
    },
  });

  console.log(response.output_text);
  ```

  ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
  from openai import OpenAI
  import os

  client = OpenAI(
      api_key=os.environ.get("ORQ_API_KEY"),
      base_url="https://my.orq.ai/v3/router",
  )

  response = client.responses.create(
      model="openai/gpt-5.4-mini",
      input="Write a marketing slogan",
      extra_body={
          "load_balancer": {
              "type": "round_robin",
              "models": [
                  {"model": "openai/gpt-5.4-mini", "weight": 0.5},
                  {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.5},
              ],
          }
      },
  )

  print(response.output_text)
  ```

  ```typescript TypeScript (Chat Completions) theme={"theme":{"light":"github-light","dark":"github-dark"}}
  import OpenAI from "openai";

  const client = new OpenAI({
    apiKey: process.env.ORQ_API_KEY,
    baseURL: "https://my.orq.ai/v3/router",
  });

  const response = await client.chat.completions.create({
    model: "openai/gpt-5.4-mini",
    messages: [{ role: "user", content: "Write a marketing slogan" }],
    load_balancer: {
      type: "round_robin",
      models: [
        { model: "openai/gpt-5.4-mini", weight: 0.5 },
        { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.5 },
      ],
    },
  });
  ```
</CodeGroup>

**How selection works:**

* Each request selects the next model in the list in turn, wrapping back to the first model after the last one. With two models, requests alternate; with three, the rotation cycles through all three.
* Weights must be included in the request but do not affect selection: round robin is purely rotational, so every model receives an equal share of traffic over time regardless of the weights.
* Round robin only selects a model per request; it does not retry or fail over a failed call.

<Note>
  Round robin selects one model per request but does not fail over when that model errors. To add failover on top of round robin, pair `load_balancer` with [Fallbacks](/ai-gateway/features/retries#fallbacks).
</Note>

## Code examples

<CodeGroup>
  ```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
  curl -X POST https://my.orq.ai/v3/router/responses \
    -H "Authorization: Bearer $ORQ_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "openai/gpt-5.4-mini",
      "input": "Write a creative marketing slogan for an eco-friendly coffee brand",
      "load_balancer": {
        "type": "weight_based",
        "models": [
          {"model": "openai/gpt-5.4-mini", "weight": 0.4},
          {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.6}
        ]
      }
    }'
  ```

  ```bash cURL (Chat Completions) theme={"theme":{"light":"github-light","dark":"github-dark"}}
  curl -X POST https://my.orq.ai/v3/router/chat/completions \
    -H "Authorization: Bearer $ORQ_API_KEY" \
    -H "Content-Type: application/json" \
    -d '{
      "model": "openai/gpt-5.4-mini",
      "messages": [
        {
          "role": "user",
          "content": "Write a creative marketing slogan for an eco-friendly coffee brand"
        }
      ],
      "load_balancer": {
        "type": "weight_based",
        "models": [
          {"model": "openai/gpt-5.4-mini", "weight": 0.4},
          {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.6}
        ]
      }
    }'
  ```

  ```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
  import OpenAI from "openai";

  const client = new OpenAI({
    apiKey: process.env.ORQ_API_KEY,
    baseURL: "https://my.orq.ai/v3/router",
  });

  const response = await client.responses.create({
    model: "openai/gpt-5.4-mini",
    input: "Write a creative marketing slogan for an eco-friendly coffee brand",
    load_balancer: {
      type: "weight_based",
      models: [
        { model: "openai/gpt-5.4-mini", weight: 0.4 },
        { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.6 },
      ],
    },
  });

  console.log(response.output_text);
  ```

  ```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
  from openai import OpenAI
  import os

  client = OpenAI(
      api_key=os.environ.get("ORQ_API_KEY"),
      base_url="https://my.orq.ai/v3/router",
  )

  response = client.responses.create(
      model="openai/gpt-5.4-mini",
      input="Write a creative marketing slogan for an eco-friendly coffee brand",
      extra_body={
          "load_balancer": {
              "type": "weight_based",
              "models": [
                  {"model": "openai/gpt-5.4-mini", "weight": 0.4},
                  {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.6},
              ],
          }
      },
  )

  print(response.output_text)
  ```

  ```typescript TypeScript (Chat Completions) theme={"theme":{"light":"github-light","dark":"github-dark"}}
  import OpenAI from "openai";

  const client = new OpenAI({
    apiKey: process.env.ORQ_API_KEY,
    baseURL: "https://my.orq.ai/v3/router",
  });

  const response = await client.chat.completions.create({
    model: "openai/gpt-5.4-mini",
    messages: [
      {
        role: "user",
        content: "Write a creative marketing slogan for an eco-friendly coffee brand",
      },
    ],
    load_balancer: {
      type: "weight_based",
      models: [
        { model: "openai/gpt-5.4-mini", weight: 0.4 },
        { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.6 },
      ],
    },
  });
  ```

  ```python Python (Chat Completions) theme={"theme":{"light":"github-light","dark":"github-dark"}}
  from openai import OpenAI
  import os

  client = OpenAI(
      api_key=os.environ.get("ORQ_API_KEY"),
      base_url="https://my.orq.ai/v3/router",
  )

  response = client.chat.completions.create(
      model="openai/gpt-5.4-mini",
      messages=[
          {
              "role": "user",
              "content": "Write a creative marketing slogan for an eco-friendly coffee brand",
          }
      ],
      extra_body={
          "load_balancer": {
              "type": "weight_based",
              "models": [
                  {"model": "openai/gpt-5.4-mini", "weight": 0.4},
                  {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.6},
              ],
          }
      },
  )
  ```
</CodeGroup>

## Monitoring

Track these metrics for optimal load balancing:

<CodeGroup>
  ```typescript Metrics tracking example theme={"theme":{"light":"github-light","dark":"github-dark"}}
  // Example monitoring setup
  const metrics = {
    requestsByModel: {}, // Count per model
    costsByModel: {}, // Cost per model
    latencyByModel: {}, // Response time per model
    errorsByModel: {}, // Error rate per model
  };
  ```
</CodeGroup>

**Key Metrics:**

* **Traffic distribution**: Actual vs expected percentages.
* **Cost per model**: Monitor spending across providers.
* **Response times**: Compare latency by model.
* **Error rates**: Track failures by provider.

With `latency_based`, response time is already the selection signal. Manual weight tuning for performance is not needed; adjust weights only to influence exploration, cold-start probing, and near-tie behavior.

## Troubleshooting

**Uneven distribution**

* Check if weights are normalized correctly.
* Verify sufficient request volume (min 100 requests for accuracy).
* Monitor over longer time periods.

**Unexpected costs**

* Track actual vs expected cost distribution.
* Monitor for expensive model overuse.
* Set up cost alerts per provider.

**Performance issues**

* Check latency differences between models.
* Monitor for provider-specific slowdowns.
* Adjust weights based on performance data.

**All traffic going to one model with `latency_based`**

* Expected once one model is consistently fastest. Ten percent of requests still explore the rest of the pool to keep their latency data fresh.
* Confirm `load_balancer.type` is set to the intended strategy if an even split was expected instead.

**Selection is slow to adapt after a deploy or restart**

* Expected. Latency history is in-memory: after a restart, every model is treated as unknown until fresh samples are collected.

## Limitations

* **Probabilistic routing**: Short-term traffic may not match exact weights.
* **Minimum volume needed**: Requires sufficient requests for statistical accuracy.
* **Response variations**: Different models may return varying output quality.
* **Cost complexity**: Managing billing across multiple providers.
* **Provider dependencies**: Requires API access to all models.
* **In-memory latency state**: With `latency_based`, latency history is in-memory and does not persist across restarts.

## Advanced weight-based usage

**Environment-specific weights:**

<CodeGroup>
  ```typescript Environment weight config theme={"theme":{"light":"github-light","dark":"github-dark"}}
  const weights = {
    development: {
      type: "weight_based",
      models: [
        { model: "openai/gpt-5.4-mini", weight: 1.0 }, // Cheap for dev
      ],
    },
    production: {
      type: "weight_based",
      models: [
        { model: "openai/gpt-5.6-sol", weight: 0.7 }, // Quality primary
        { model: "anthropic/claude-sonnet-5", weight: 0.3 }, // Backup
      ],
    },
  };
  ```
</CodeGroup>

**Dynamic weight adjustment:**

<CodeGroup>
  ```typescript Dynamic weight calculation theme={"theme":{"light":"github-light","dark":"github-dark"}}
  // Adjust weights based on performance
  const calculateWeight = (latency: number, cost: number, quality: number) =>
    quality / (latency * cost);

  const adjustWeights = (models) => ({
    type: "weight_based",
    models: models.map((model) => ({
      model: model.model,
      weight: calculateWeight(model.latency, model.cost, model.quality),
    })),
  });
  ```
</CodeGroup>

**With other features:**

<CodeGroup>
  ```json JSON theme={"theme":{"light":"github-light","dark":"github-dark"}}
  {
    "model": "openai/gpt-5.6-sol",
    "load_balancer": {
      "type": "weight_based",
      "models": [
        { "model": "openai/gpt-5.6-sol", "weight": 0.6 },
        { "model": "anthropic/claude-sonnet-5", "weight": 0.4 }
      ]
    },
    "retry": { "count": 2, "on_codes": [429] },
    "timeout": { "call_timeout": 15000 }
  }
  ```
</CodeGroup>