Load balancing across providers

Use Cases

Distributing traffic across multiple provider accounts to stay within per-key rate limits.
A/B testing providers by routing a configurable percentage of traffic to each.
Spreading traffic across multiple providers to reduce blast radius from a single provider outage.
Maximizing throughput when one provider’s capacity is a bottleneck.
Minimizing response time by routing to the fastest available model.

Quick Start

Distribute requests across multiple providers using weighted routing.

curl -X POST https://api.orq.ai/v3/router/responses \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "input": "Write a marketing slogan",
    "load_balancer": {
      "type": "weight_based",
      "models": [
        {"model": "openai/gpt-5-mini", "weight": 0.7},
        {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.3}
      ]
    }
  }'

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.ORQ_API_KEY,
  baseURL: "https://api.orq.ai/v3/router",
});

const response = await client.responses.create({
  model: "openai/gpt-4o-mini",
  input: "Write a marketing slogan",
  load_balancer: {
    type: "weight_based",
    models: [
      { model: "openai/gpt-5-mini", weight: 0.7 },
      { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.3 },
    ],
  },
});

console.log(response.output_text);

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ.get("ORQ_API_KEY"),
    base_url="https://api.orq.ai/v3/router",
)

response = client.responses.create(
    model="openai/gpt-4o-mini",
    input="Write a marketing slogan",
    extra_body={
        "load_balancer": {
            "type": "weight_based",
            "models": [
                {"model": "openai/gpt-5-mini", "weight": 0.7},
                {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.3},
            ],
        }
    },
)

print(response.output_text)

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.ORQ_API_KEY,
  baseURL: "https://api.orq.ai/v3/router",
});

const response = await client.chat.completions.create({
  model: "openai/gpt-4o-mini",
  messages: [{ role: "user", content: "Write a marketing slogan" }],
  load_balancer: {
    type: "weight_based",
    models: [
      { model: "openai/gpt-5-mini", weight: 0.7 },
      { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.3 },
    ],
  },
});

Configuration

Parameter	Type	Required	Description
`load_balancer`	Object	Yes	Load balancer configuration (top-level)
`load_balancer.type`	string	Yes	Strategy type (`weight_based`, `round_robin`, or `latency_based`)
`load_balancer.models`	Array	Yes	List of models with weights
`models[].model`	string	Yes	Model identifier
`models[].weight`	number	No	Relative weight (0.001 - 1.0, default 0.5)

Weight Calculation:

Weights are normalized: [0.4, 0.8] → [33%, 67%].
Higher weight = more traffic.
Minimum weight: 0.001.
Default weight: 0.5.

A matching Routing Rule with its own target models configured overwrites this load_balancer parameter entirely, regardless of what the request sends. The Routing Rule’s own strategy and models are used instead; the two configurations are never merged.

load_balancer selects one model per request. It does not retry a failed call against another model in the pool; it only records the failure so future selections avoid it. The top-level model field is used as-is only if load_balancer fails to produce a selection (for example an empty models list). To fail over to a different model when a call errors, pair load_balancer with Fallbacks.

Weight-based routing

Split traffic across models by percentage weights instead of latency or a fixed rotation.

Patterns

// Equal distribution
load_balancer: {
  type: "weight_based",
  models: [
    { model: "openai/gpt-4o", weight: 1.0 },
    { model: "anthropic/claude-sonnet-4-6", weight: 1.0 },
  ],
}

// Cost optimization (cheap model primary)
load_balancer: {
  type: "weight_based",
  models: [
    { model: "openai/gpt-5-mini", weight: 0.8 },
    { model: "openai/gpt-4o", weight: 0.2 },
  ],
}

// A/B testing
load_balancer: {
  type: "weight_based",
  models: [
    { model: "current-model", weight: 0.9 },
    { model: "experimental-model", weight: 0.1 },
  ],
}

// Multi-provider redundancy
load_balancer: {
  type: "weight_based",
  models: [
    { model: "openai/gpt-4o", weight: 0.5 },
    { model: "anthropic/claude-sonnet-4-6", weight: 0.3 },
    { model: "azure/gpt-4o", weight: 0.2 },
  ],
}

Use cases

Scenario	Approach	Example
Cost optimization	Heavy on cheaper models	80% GPT-3.5, 20% GPT-4
Performance testing	Small traffic to new model	95% current, 5% experimental
Provider redundancy	Split across providers	60% OpenAI, 40% Anthropic
Capacity management	Distribute during peaks	Even split across models

Latency-based routing

Route each request to the model with the lowest recently observed latency, instead of a fixed traffic split.

curl -X POST https://api.orq.ai/v3/router/responses \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "input": "Write a marketing slogan",
    "load_balancer": {
      "type": "latency_based",
      "models": [
        {"model": "openai/gpt-5-mini", "weight": 0.5},
        {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.5}
      ]
    }
  }'

How selection works:

Each AI Gateway replica tracks its own in-memory latency history per provider/model as an exponentially weighted moving average (EWMA): a running average where each new sample blends in at 30% weight against 70% for the existing average, so recent calls shift the average faster than older ones without discarding history entirely. This history is not shared between replicas and is lost when a replica restarts.
With fresh data (samples from the last 5 minutes) for every configured model, the lowest-latency model is selected. A model with no data, or stale data, is probed again instead of being written off.
Ten percent of selections explore the full pool by configured weight regardless of current latency, keeping measurements fresh. A failed call is penalized (the greater of twice its duration or 1,000 ms) so a fast failure doesn’t look like a fast success.
Configured weights decide exploration, probing, and near-ties (latencies within 0.5 ms of each other). With fresh, non-tied data, the lowest latency always wins regardless of weight.

Full selection flow

Code examples

curl -X POST https://api.orq.ai/v3/router/responses \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "input": "Write a creative marketing slogan for an eco-friendly coffee brand",
    "load_balancer": {
      "type": "weight_based",
      "models": [
        {"model": "openai/gpt-5-mini", "weight": 0.4},
        {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.6}
      ]
    }
  }'

curl -X POST https://api.orq.ai/v3/router/chat/completions \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o-mini",
    "messages": [
      {
        "role": "user",
        "content": "Write a creative marketing slogan for an eco-friendly coffee brand"
      }
    ],
    "load_balancer": {
      "type": "weight_based",
      "models": [
        {"model": "openai/gpt-5-mini", "weight": 0.4},
        {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.6}
      ]
    }
  }'

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.ORQ_API_KEY,
  baseURL: "https://api.orq.ai/v3/router",
});

const response = await client.responses.create({
  model: "openai/gpt-4o-mini",
  input: "Write a creative marketing slogan for an eco-friendly coffee brand",
  load_balancer: {
    type: "weight_based",
    models: [
      { model: "openai/gpt-5-mini", weight: 0.4 },
      { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.6 },
    ],
  },
});

console.log(response.output_text);

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ.get("ORQ_API_KEY"),
    base_url="https://api.orq.ai/v3/router",
)

response = client.responses.create(
    model="openai/gpt-4o-mini",
    input="Write a creative marketing slogan for an eco-friendly coffee brand",
    extra_body={
        "load_balancer": {
            "type": "weight_based",
            "models": [
                {"model": "openai/gpt-5-mini", "weight": 0.4},
                {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.6},
            ],
        }
    },
)

print(response.output_text)

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.ORQ_API_KEY,
  baseURL: "https://api.orq.ai/v3/router",
});

const response = await client.chat.completions.create({
  model: "openai/gpt-4o-mini",
  messages: [
    {
      role: "user",
      content: "Write a creative marketing slogan for an eco-friendly coffee brand",
    },
  ],
  load_balancer: {
    type: "weight_based",
    models: [
      { model: "openai/gpt-5-mini", weight: 0.4 },
      { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.6 },
    ],
  },
});

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ.get("ORQ_API_KEY"),
    base_url="https://api.orq.ai/v3/router",
)

response = client.chat.completions.create(
    model="openai/gpt-4o-mini",
    messages=[
        {
            "role": "user",
            "content": "Write a creative marketing slogan for an eco-friendly coffee brand",
        }
    ],
    extra_body={
        "load_balancer": {
            "type": "weight_based",
            "models": [
                {"model": "openai/gpt-5-mini", "weight": 0.4},
                {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.6},
            ],
        }
    },
)

Monitoring

Track these metrics for optimal load balancing:

// Example monitoring setup
const metrics = {
  requestsByModel: {}, // Count per model
  costsByModel: {}, // Cost per model
  latencyByModel: {}, // Response time per model
  errorsByModel: {}, // Error rate per model
};

Key Metrics:

Traffic distribution: Actual vs expected percentages.
Cost per model: Monitor spending across providers.
Response times: Compare latency by model.
Error rates: Track failures by provider.

With latency_based, response time is already the selection signal. Manual weight tuning for performance is not needed; adjust weights only to influence exploration, cold-start probing, and near-tie behavior.

Troubleshooting

Uneven distribution

Check if weights are normalized correctly.
Verify sufficient request volume (min 100 requests for accuracy).
Monitor over longer time periods. Unexpected costs
Track actual vs expected cost distribution.
Monitor for expensive model overuse.
Set up cost alerts per provider. Performance issues
Check latency differences between models.
Monitor for provider-specific slowdowns.
Adjust weights based on performance data. All traffic going to one model with latency_based
Expected once one model is consistently fastest. Ten percent of requests still explore the rest of the pool to keep their latency data fresh.
Confirm load_balancer.type is set to the intended strategy if an even split was expected instead.

Selection is slow to adapt after a deploy or restart

Expected. Latency history is in-memory and local to each replica. A restarted replica treats every model as unknown until it collects fresh samples.

Limitations

Probabilistic routing: Short-term traffic may not match exact weights.
Minimum volume needed: Requires sufficient requests for statistical accuracy.
Response variations: Different models may return varying output quality.
Cost complexity: Managing billing across multiple providers.
Provider dependencies: Requires API access to all models.
Per-replica latency state: With latency_based, latency history is in-memory and local to each AI Gateway replica. It is not shared across replicas and does not persist across restarts.

Advanced weight-based usage

Environment-specific weights:

const weights = {
  development: {
    type: "weight_based",
    models: [
      { model: "openai/gpt-5-mini", weight: 1.0 }, // Cheap for dev
    ],
  },
  production: {
    type: "weight_based",
    models: [
      { model: "openai/gpt-4o", weight: 0.7 }, // Quality primary
      { model: "anthropic/claude-sonnet-4-6", weight: 0.3 }, // Backup
    ],
  },
};

Dynamic weight adjustment:

// Adjust weights based on performance
const calculateWeight = (latency: number, cost: number, quality: number) =>
  quality / (latency * cost);

const adjustWeights = (models) => ({
  type: "weight_based",
  models: models.map((model) => ({
    model: model.model,
    weight: calculateWeight(model.latency, model.cost, model.quality),
  })),
});

With other features:

{
  "model": "openai/gpt-4o",
  "load_balancer": {
    "type": "weight_based",
    "models": [
      { "model": "openai/gpt-4o", "weight": 0.6 },
      { "model": "anthropic/claude-sonnet-4-6", "weight": 0.4 }
    ]
  },
  "retry": { "count": 2, "on_codes": [429] },
  "timeout": { "call_timeout": 15000 }
}

Get Started

Models & Providers

Configuration

Observability

AI Gateway Features

Quick Start

Configuration

Weight-based routing

Patterns

Use cases

Latency-based routing

See also: Organization-level load balancing

Code examples

Monitoring

Troubleshooting

Limitations

Advanced weight-based usage

​Quick Start

​Configuration

​Weight-based routing

​Patterns

​Use cases

​Latency-based routing

See also: Organization-level load balancing

​Code examples

​Monitoring

​Troubleshooting

​Limitations

​Advanced weight-based usage

Quick Start

Configuration

Weight-based routing

Patterns

Use cases

Latency-based routing

Code examples

Monitoring

Troubleshooting

Limitations

Advanced weight-based usage