Load balancing across providers

Use Cases

Distributing traffic across multiple provider accounts to stay within per-key rate limits.
A/B testing providers by routing a configurable percentage of traffic to each.
Spreading traffic across multiple providers to reduce blast radius from a single provider outage.
Maximizing throughput when one provider’s capacity is a bottleneck.
Minimizing response time by routing to the fastest available model.

Quick Start

Distribute requests across multiple providers using weighted routing.

curl -X POST https://api.orq.ai/v3/router/responses \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-5.4-mini",
    "input": "Write a marketing slogan",
    "load_balancer": {
      "type": "weight_based",
      "models": [
        {"model": "openai/gpt-5.4-mini", "weight": 0.7},
        {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.3}
      ]
    }
  }'

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.ORQ_API_KEY,
  baseURL: "https://api.orq.ai/v3/router",
});

const response = await client.responses.create({
  model: "openai/gpt-5.4-mini",
  input: "Write a marketing slogan",
  load_balancer: {
    type: "weight_based",
    models: [
      { model: "openai/gpt-5.4-mini", weight: 0.7 },
      { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.3 },
    ],
  },
});

console.log(response.output_text);

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ.get("ORQ_API_KEY"),
    base_url="https://api.orq.ai/v3/router",
)

response = client.responses.create(
    model="openai/gpt-5.4-mini",
    input="Write a marketing slogan",
    extra_body={
        "load_balancer": {
            "type": "weight_based",
            "models": [
                {"model": "openai/gpt-5.4-mini", "weight": 0.7},
                {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.3},
            ],
        }
    },
)

print(response.output_text)

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.ORQ_API_KEY,
  baseURL: "https://api.orq.ai/v3/router",
});

const response = await client.chat.completions.create({
  model: "openai/gpt-5.4-mini",
  messages: [{ role: "user", content: "Write a marketing slogan" }],
  load_balancer: {
    type: "weight_based",
    models: [
      { model: "openai/gpt-5.4-mini", weight: 0.7 },
      { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.3 },
    ],
  },
});

Configuration

Parameter	Type	Required	Description
`load_balancer`	Object	Yes	Load balancer configuration (top-level)
`load_balancer.type`	string	Yes	Strategy type (`weight_based`, `round_robin`, or `latency_based`)
`load_balancer.models`	Array	Yes	List of models with weights
`models[].model`	string	Yes	Model identifier
`models[].weight`	number	Yes	Weight assigned to this model (0.001 - 1.0)

Weight Calculation:

Weights are normalized: [0.4, 0.8] → [33%, 67%].
Higher weight = more traffic.
Minimum weight: 0.001.
Weight is ignored for round robin selection; every model in the list still receives an equal share of traffic.

A matching Routing Rule with its own target models configured overwrites this load_balancer parameter entirely, regardless of what the request sends. The Routing Rule’s own strategy and models are used instead; the two configurations are never merged.

load_balancer selects one model per request. It does not retry a failed call against another model in the pool. The top-level model is used as-is only when load_balancer cannot produce a selection, for example an empty models list. To fail over when a call errors, pair load_balancer with Fallbacks.

Latency-based routing

Route each request to the model with the lowest recently observed latency, instead of a fixed traffic split.

curl -X POST https://api.orq.ai/v3/router/responses \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-5.4-mini",
    "input": "Write a marketing slogan",
    "load_balancer": {
      "type": "latency_based",
      "models": [
        {"model": "openai/gpt-5.4-mini", "weight": 0.5},
        {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.5}
      ]
    }
  }'

How selection works:

Latency is tracked per model as a running average that weights recent calls more heavily, so the selection adapts quickly when a provider speeds up or slows down.
With fresh data (samples from the last 5 minutes) for every configured model, the lowest-latency model is selected. A model with no data, or stale data, is probed again instead of being written off.
Ten percent of selections explore the full pool by configured weight to keep latency measurements fresh. Failed calls are penalized so a fast failure does not look like a fast success.
Configured weights decide exploration and near-ties: with the two fastest models within 0.5 ms of each other, the higher weight wins. With clear data, the lowest latency always wins.

Full selection flow

Weight-based routing

Split traffic across models by percentage weights instead of latency or a fixed rotation.

Patterns

// Equal distribution
load_balancer: {
  type: "weight_based",
  models: [
    { model: "openai/gpt-5.6-sol", weight: 1.0 },
    { model: "anthropic/claude-sonnet-5", weight: 1.0 },
  ],
}

// Cost optimization (cheap model primary)
load_balancer: {
  type: "weight_based",
  models: [
    { model: "openai/gpt-5.4-mini", weight: 0.8 },
    { model: "openai/gpt-5.6-sol", weight: 0.2 },
  ],
}

// A/B testing
load_balancer: {
  type: "weight_based",
  models: [
    { model: "current-model", weight: 0.9 },
    { model: "experimental-model", weight: 0.1 },
  ],
}

// Multi-provider redundancy
load_balancer: {
  type: "weight_based",
  models: [
    { model: "openai/gpt-5.6-sol", weight: 0.5 },
    { model: "anthropic/claude-sonnet-5", weight: 0.3 },
    { model: "azure/gpt-5.6-sol", weight: 0.2 },
  ],
}

Use cases

Scenario	Approach	Example
Cost optimization	Heavy on cheaper models	80% GPT-5.4 Mini, 20% GPT-5.6 Sol
Performance testing	Small traffic to new model	95% current, 5% experimental
Provider redundancy	Split across providers	60% OpenAI, 40% Anthropic
Capacity management	Distribute during peaks	Even split across models

Round Robin routing

Rotate through the configured models evenly, one request at a time, instead of splitting traffic by weight or latency.

curl -X POST https://api.orq.ai/v3/router/responses \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-5.4-mini",
    "input": "Write a marketing slogan",
    "load_balancer": {
      "type": "round_robin",
      "models": [
        {"model": "openai/gpt-5.4-mini", "weight": 0.5},
        {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.5}
      ]
    }
  }'

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.ORQ_API_KEY,
  baseURL: "https://api.orq.ai/v3/router",
});

const response = await client.responses.create({
  model: "openai/gpt-5.4-mini",
  input: "Write a marketing slogan",
  load_balancer: {
    type: "round_robin",
    models: [
      { model: "openai/gpt-5.4-mini", weight: 0.5 },
      { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.5 },
    ],
  },
});

console.log(response.output_text);

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ.get("ORQ_API_KEY"),
    base_url="https://api.orq.ai/v3/router",
)

response = client.responses.create(
    model="openai/gpt-5.4-mini",
    input="Write a marketing slogan",
    extra_body={
        "load_balancer": {
            "type": "round_robin",
            "models": [
                {"model": "openai/gpt-5.4-mini", "weight": 0.5},
                {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.5},
            ],
        }
    },
)

print(response.output_text)

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.ORQ_API_KEY,
  baseURL: "https://api.orq.ai/v3/router",
});

const response = await client.chat.completions.create({
  model: "openai/gpt-5.4-mini",
  messages: [{ role: "user", content: "Write a marketing slogan" }],
  load_balancer: {
    type: "round_robin",
    models: [
      { model: "openai/gpt-5.4-mini", weight: 0.5 },
      { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.5 },
    ],
  },
});

How selection works:

Each request selects the next model in the list in turn, wrapping back to the first model after the last one. With two models, requests alternate; with three, the rotation cycles through all three.
Weights must be included in the request but do not affect selection: round robin is purely rotational, so every model receives an equal share of traffic over time regardless of the weights.
Round robin only selects a model per request; it does not retry or fail over a failed call.

Round robin selects one model per request but does not fail over when that model errors. To add failover on top of round robin, pair load_balancer with Fallbacks.

Code examples

curl -X POST https://api.orq.ai/v3/router/responses \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-5.4-mini",
    "input": "Write a creative marketing slogan for an eco-friendly coffee brand",
    "load_balancer": {
      "type": "weight_based",
      "models": [
        {"model": "openai/gpt-5.4-mini", "weight": 0.4},
        {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.6}
      ]
    }
  }'

curl -X POST https://api.orq.ai/v3/router/chat/completions \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-5.4-mini",
    "messages": [
      {
        "role": "user",
        "content": "Write a creative marketing slogan for an eco-friendly coffee brand"
      }
    ],
    "load_balancer": {
      "type": "weight_based",
      "models": [
        {"model": "openai/gpt-5.4-mini", "weight": 0.4},
        {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.6}
      ]
    }
  }'

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.ORQ_API_KEY,
  baseURL: "https://api.orq.ai/v3/router",
});

const response = await client.responses.create({
  model: "openai/gpt-5.4-mini",
  input: "Write a creative marketing slogan for an eco-friendly coffee brand",
  load_balancer: {
    type: "weight_based",
    models: [
      { model: "openai/gpt-5.4-mini", weight: 0.4 },
      { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.6 },
    ],
  },
});

console.log(response.output_text);

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ.get("ORQ_API_KEY"),
    base_url="https://api.orq.ai/v3/router",
)

response = client.responses.create(
    model="openai/gpt-5.4-mini",
    input="Write a creative marketing slogan for an eco-friendly coffee brand",
    extra_body={
        "load_balancer": {
            "type": "weight_based",
            "models": [
                {"model": "openai/gpt-5.4-mini", "weight": 0.4},
                {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.6},
            ],
        }
    },
)

print(response.output_text)

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.ORQ_API_KEY,
  baseURL: "https://api.orq.ai/v3/router",
});

const response = await client.chat.completions.create({
  model: "openai/gpt-5.4-mini",
  messages: [
    {
      role: "user",
      content: "Write a creative marketing slogan for an eco-friendly coffee brand",
    },
  ],
  load_balancer: {
    type: "weight_based",
    models: [
      { model: "openai/gpt-5.4-mini", weight: 0.4 },
      { model: "anthropic/claude-haiku-4-5-20251001", weight: 0.6 },
    ],
  },
});

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ.get("ORQ_API_KEY"),
    base_url="https://api.orq.ai/v3/router",
)

response = client.chat.completions.create(
    model="openai/gpt-5.4-mini",
    messages=[
        {
            "role": "user",
            "content": "Write a creative marketing slogan for an eco-friendly coffee brand",
        }
    ],
    extra_body={
        "load_balancer": {
            "type": "weight_based",
            "models": [
                {"model": "openai/gpt-5.4-mini", "weight": 0.4},
                {"model": "anthropic/claude-haiku-4-5-20251001", "weight": 0.6},
            ],
        }
    },
)

Monitoring

Track these metrics for optimal load balancing:

// Example monitoring setup
const metrics = {
  requestsByModel: {}, // Count per model
  costsByModel: {}, // Cost per model
  latencyByModel: {}, // Response time per model
  errorsByModel: {}, // Error rate per model
};

Key Metrics:

Traffic distribution: Actual vs expected percentages.
Cost per model: Monitor spending across providers.
Response times: Compare latency by model.
Error rates: Track failures by provider.

With latency_based, response time is already the selection signal. Manual weight tuning for performance is not needed; adjust weights only to influence exploration, cold-start probing, and near-tie behavior.

Troubleshooting

Uneven distribution

Check if weights are normalized correctly.
Verify sufficient request volume (min 100 requests for accuracy).
Monitor over longer time periods.

Unexpected costs

Track actual vs expected cost distribution.
Monitor for expensive model overuse.
Set up cost alerts per provider.

Performance issues

Check latency differences between models.
Monitor for provider-specific slowdowns.
Adjust weights based on performance data.

All traffic going to one model with latency_based

Expected once one model is consistently fastest. Ten percent of requests still explore the rest of the pool to keep their latency data fresh.
Confirm load_balancer.type is set to the intended strategy if an even split was expected instead.

Selection is slow to adapt after a deploy or restart

Expected. Latency history is in-memory: after a restart, every model is treated as unknown until fresh samples are collected.

Limitations

Probabilistic routing: Short-term traffic may not match exact weights.
Minimum volume needed: Requires sufficient requests for statistical accuracy.
Response variations: Different models may return varying output quality.
Cost complexity: Managing billing across multiple providers.
Provider dependencies: Requires API access to all models.
In-memory latency state: With latency_based, latency history is in-memory and does not persist across restarts.

Advanced weight-based usage

Environment-specific weights:

const weights = {
  development: {
    type: "weight_based",
    models: [
      { model: "openai/gpt-5.4-mini", weight: 1.0 }, // Cheap for dev
    ],
  },
  production: {
    type: "weight_based",
    models: [
      { model: "openai/gpt-5.6-sol", weight: 0.7 }, // Quality primary
      { model: "anthropic/claude-sonnet-5", weight: 0.3 }, // Backup
    ],
  },
};

Dynamic weight adjustment:

// Adjust weights based on performance
const calculateWeight = (latency: number, cost: number, quality: number) =>
  quality / (latency * cost);

const adjustWeights = (models) => ({
  type: "weight_based",
  models: models.map((model) => ({
    model: model.model,
    weight: calculateWeight(model.latency, model.cost, model.quality),
  })),
});

With other features:

{
  "model": "openai/gpt-5.6-sol",
  "load_balancer": {
    "type": "weight_based",
    "models": [
      { "model": "openai/gpt-5.6-sol", "weight": 0.6 },
      { "model": "anthropic/claude-sonnet-5", "weight": 0.4 }
    ]
  },
  "retry": { "count": 2, "on_codes": [429] },
  "timeout": { "call_timeout": 15000 }
}

Overview

APIs

Routing

Security & Privacy

Model Capabilities

Analytics & Monitoring

Quick Start

Configuration

Latency-based routing

See also: Organization-level load balancing

Weight-based routing

Patterns

Use cases

Round Robin routing

Code examples

Monitoring

Troubleshooting

Limitations

Advanced weight-based usage

​Quick Start

​Configuration

​Latency-based routing

See also: Organization-level load balancing

​Weight-based routing

​Patterns

​Use cases

​Round Robin routing

​Code examples

​Monitoring

​Troubleshooting

​Limitations

​Advanced weight-based usage

Quick Start

Configuration

Latency-based routing

Weight-based routing

Patterns

Use cases

Round Robin routing

Code examples

Monitoring

Troubleshooting

Limitations

Advanced weight-based usage