Session Affinity

A Smart Router picks a model per request. In a multi-turn conversation that means every turn is scored again and can land on a different model, which resets the provider prompt cache. Session affinity pins a conversation to the model chosen on its first turn. Later turns reuse that model with no scoring call on the request path, and keep the cache warm. Session affinity is off by default and configured per router. Only requests that carry a thread id or, on the Responses endpoint, a conversation id take part. Everything else is routed per request as before.

Use Cases

Chat assistants and copilots where consecutive turns share a long system prompt or tool set and benefit from the provider prompt cache.
Agent loops that call the router several times per user turn and should not change models halfway through.
Support conversations where the tone and level of detail should stay consistent from the first reply to the last.

Enable Session Affinity

In the UI

Open the Smart Router in AI Gateway and edit it, or create a new one.
Under Session affinity, turn on Pin conversations to one model.
Set Pin for to the idle window: 1, 5, 10, 30, or 60 minutes. The default is 10 minutes.
Click Save changes, or Create for a new router.

Through the API

Send session_affinity when creating or updating a Smart Router. ttl_seconds is the idle window in seconds, between 60 and 3600, and defaults to 600 when omitted. On update the object is replaced as a whole; omitting ttl_seconds resets the window to the default of 600.

curl -X PATCH https://api.orq.ai/v2/smart-routers/$SMART_ROUTER_ID \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "session_affinity": { "enabled": true, "ttl_seconds": 600 }
  }'

const response = await fetch(
  `https://api.orq.ai/v2/smart-routers/${process.env.SMART_ROUTER_ID}`,
  {
    method: "PATCH",
    headers: {
      Authorization: `Bearer ${process.env.ORQ_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      session_affinity: { enabled: true, ttl_seconds: 600 },
    }),
  },
);

console.log(await response.json());

import os
import requests

response = requests.patch(
    f"https://api.orq.ai/v2/smart-routers/{os.environ['SMART_ROUTER_ID']}",
    headers={"Authorization": f"Bearer {os.environ['ORQ_API_KEY']}"},
    json={"session_affinity": {"enabled": True, "ttl_seconds": 600}},
)

print(response.json())

The same object is accepted on POST /v2/smart-routers. See the Smart Routers API reference.

Seed the Pin

The pin is keyed on an id the caller sends. Nothing is derived from the message content.

Endpoint	Seed	How to send it
`/v3/router/chat/completions`	Thread id	`thread.id` in the body, or the `X-ORQ-THREAD-ID` header
`/v3/anthropic/v1/messages`	Thread id	`thread.id` in the body, or the `X-ORQ-THREAD-ID` header
`/v3/router/responses`	Thread id, else conversation id	`thread.id` or `X-ORQ-THREAD-ID`; `conversation.id` when no thread id is sent
`/v2/deployments/invoke`	Thread id	`thread.id` in the body, or the `X-ORQ-THREAD-ID` header
Agents	Thread id, else conversation id	Same as the Responses endpoint

When both are present the body wins over the header. A thread id and a conversation id with the same value produce two independent pins. See Request Metadata for the thread field and headers.

Quick Start

Call the Smart Router twice with the same thread id. The first turn is scored and returns X-Orq-Session-Affinity: miss; the second turn reuses the pinned model and returns hit.

for turn in "What does the refund policy say about digital goods?" "And for physical goods?"; do
  curl -s -D - -o /dev/null -X POST https://my.orq.ai/v3/router/chat/completions \
    -H "Authorization: Bearer $ORQ_API_KEY" \
    -H "Content-Type: application/json" \
    -d "{
      \"model\": \"workspace_name@orq/my-smart-router\",
      \"messages\": [{ \"role\": \"user\", \"content\": \"$turn\" }],
      \"thread\": { \"id\": \"conversation-abc123\" }
    }" | grep -i x-orq-session-affinity
done

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.ORQ_API_KEY,
  baseURL: "https://my.orq.ai/v3/router",
});

const thread = { id: "conversation-abc123" };

for (const content of [
  "What does the refund policy say about digital goods?",
  "And for physical goods?",
]) {
  const { data, response } = await client.chat.completions
    .create({
      model: "workspace_name@orq/my-smart-router",
      messages: [{ role: "user", content }],
      // @ts-expect-error Orq.ai extension
      thread,
    })
    .withResponse();

  console.log(response.headers.get("x-orq-session-affinity"), data.model);
}

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ.get("ORQ_API_KEY"),
    base_url="https://my.orq.ai/v3/router",
)

for content in [
    "What does the refund policy say about digital goods?",
    "And for physical goods?",
]:
    raw = client.chat.completions.with_raw_response.create(
        model="workspace_name@orq/my-smart-router",
        messages=[{"role": "user", "content": content}],
        extra_body={"thread": {"id": "conversation-abc123"}},
    )
    print(raw.headers.get("x-orq-session-affinity"), raw.parse().model)

The model field in the response names the concrete model on both turns.

How a Pin Is Decided

The first turn of a thread is scored and routed as usual. The pin is written only after that turn’s first provider call succeeds, so a failed or retried first turn never pins a model. A turn the scorer could not classify with confidence is served but not pinned; the next turn tries again. On a hit, the request goes straight to the pinned model with no scoring call on the request path, and the idle window is refreshed. Easy and Medium hits still trigger one background scoring call for escalation; Hard hits make none.

Escalation

A thread that starts easy can turn hard. On every hit with an Easy or Medium pin, the router re-scores the turn in the background. When the new score ranks in a higher band, the pin moves up to that band’s model and stays there. A pin never moves down: a downgrade would reset the provider cache for a small saving. Hard pins are never re-scored. The escalation is a compare-and-set write that keeps the pin’s remaining lifetime, so two turns racing on one thread cannot move the decision backwards.

Pin Lifetime

The idle window is sliding. Every hit refreshes it, so an active conversation keeps its model for as long as it stays active, and a thread that stays quiet longer than the window is routed fresh on its next turn. There is no absolute ceiling. A pin is dropped, and the request routed per request instead, when:

The pinned model fails on the first provider call of a turn.
The pinned model is no longer in the router’s pool, or no longer callable in the workspace.
The stored pin cannot be parsed.

When the pin store cannot be read, the pin is ignored for that turn and the turn is routed fresh. A dropped or ignored pin never fails the request.

Where It Applies

Surface	Session affinity	`X-Orq-Session-Affinity` header
`/v3/router/chat/completions`	Yes	`hit` or `miss`
`/v3/anthropic/v1/messages`	Yes	`hit` or `miss`
`/v3/router/responses`	Yes	`hit` or `miss`
`/v2/deployments/invoke` (JSON and streaming)	Yes, when the variant’s model is a Smart Router	`hit` or `miss`
Agents whose primary model is a Smart Router	Yes	Not set
Background responses (`background: true`)	Reads an existing pin only	`hit` or `miss`

On the Responses and Messages endpoints, a turn that runs a tool loop settles the pin on its first provider call only. Sub-agents pin under their own router and their own key. Scheduled agent runs carry no thread or conversation id and are routed per request.

Observability

Every routed request records the decision on its span:

Attribute	Meaning
`orq.session_affinity.enabled`	Session affinity is on for this router
`orq.session_affinity.seed`	`thread` or `conversation`
`orq.session_affinity.hit`	The turn reused a pin
`orq.session_affinity.ttl_seconds`	The router’s idle window
`orq.session_affinity.escalated`	The pin moved up on this turn
`orq.session_affinity.escalated_from`, `orq.session_affinity.escalated_to`	The bands involved

The escalation attributes are recorded on a separate auto-router.escalation span in the same trace, because the re-score runs after the response is sent. The metrics router.session_affinity.hits, router.session_affinity.misses, router.session_affinity.dissolves, and router.session_affinity.escalations count the same events per router. See Traces.

A pin holds a model name, a band, and a timestamp. No message content or user data is stored with it.

Limitations

Limitation	Impact	Workaround
Needs a thread id or conversation id	Requests without either are routed per request	Send `thread.id`, the `X-ORQ-THREAD-ID` header, or `conversation.id` on the Responses endpoint
Pins are per router	A load balancer in front of several Smart Routers reshuffles requests every turn	Put session affinity on one Smart Router and call it directly
Pins are per API key and identity	The same thread called with a different API key or identity gets its own pin	Keep one API key and identity per conversation
Background responses	`background: true` requests reuse a pin but never write or drop one	Start the thread with a foreground request
Agents return no header	The decision is visible on the span, not on the response	Filter traces on `orq.session_affinity.hit`

Overview

APIs

MCP Portal

Routing

Security & Privacy

Model Capabilities

Server Tools

Common Scenarios

Use Cases

Enable Session Affinity

In the UI

Through the API

Seed the Pin

Quick Start

How a Pin Is Decided

Escalation

Pin Lifetime

Where It Applies

Observability

Limitations

​Use Cases

​Enable Session Affinity

​In the UI

​Through the API

​Seed the Pin

​Quick Start

​How a Pin Is Decided

​Escalation

​Pin Lifetime

​Where It Applies

​Observability

​Limitations

Use Cases

Enable Session Affinity

In the UI

Through the API

Seed the Pin

Quick Start

How a Pin Is Decided

Escalation

Pin Lifetime

Where It Applies

Observability

Limitations