Prompt caching for reduced token costs

Use Cases

Reusing long system prompts across many requests to cut input token costs.
Referencing large documents or codebases without re-sending them every call.
Multi-turn conversations with a large, stable context that doesn’t change between turns.
RAG pipelines where the same retrieved context is shared across many user queries.

Overview

Prompt Caching is a provider-level feature that caches prompts so that repeated requests are charged at a reduced rate. This is most effective when your requests share a large, stable prefix:

a long system prompt.
a reference document.
a tool definition list. Unlike Response Caching, which serves a stored response for identical requests, Prompt Caching still calls the model on every request, at a reduced cost. Both can be used together.

How caching is enabled and what gets cached varies by provider. See the provider sections below.

Anthropic

Prompt caching on Anthropic models requires explicit opt-in via cache_control markers on individual message parts. A breakpoint marks the end of a cacheable prefix: the provider caches everything up to and including the marked block, and later requests that share that prefix read the cached portion instead of reprocessing it. On the Anthropic-compatible endpoint, breakpoints pass through exactly as in the native Anthropic API; see the Anthropic Messages API for endpoint details and a multi-turn example.

Supported models

All current Claude models, including Claude Fable 5 and the Opus, Sonnet, and Haiku families.

Enabling caching

Add a cache_control object to any message part you want to mark as cacheable:

{
  "cache_control": { "type": "ephemeral" }
}

"ephemeral" is the only supported type. You can place it on:

System message text parts.
User message text parts.
User message images, documents, and files (including PDFs).
Tool definitions.
Tool result content.

Minimum token thresholds

Caching only activates once the marked content exceeds a minimum token count. Requests below the threshold are processed normally at full cost.

Model	Minimum tokens
Claude Opus 5, Fable 5	512
Claude Opus 4.8, Sonnet 5, Sonnet 4.6, Sonnet 4.5, Opus 4.1, Opus 4, Sonnet 4, Sonnet 3.7	1,024
Claude Opus 4.7, Haiku 3.5, Haiku 3	2,048
Claude Opus 4.6, Opus 4.5, Haiku 4.5	4,096

Cache TTL

The ttl parameter controls how long cached content persists before expiring.

Value	Duration
`"5m"` (default)	5 minutes from last use
`"1h"`	1 hour

{
  "cache_control": {
    "type": "ephemeral",
    "ttl": "1h"
  }
}

Cache reads are billed at a fraction of the regular input token price. Cache writes carry a premium: 1.25x the input price for the 5-minute TTL and 2x for the 1-hour TTL. Caching pays off from the second request onward on a shared prefix.

Multi-turn conversations

Default: mark the system prompt. Put cache_control on the system text part and send the full input on every request, including any conversation history the client keeps. The system prompt is written once and read on every later turn, no matter what the user says. Every other example on this page uses this form. When continuing with previous_response_id, use the top-level parameter instead. The gateway rebuilds the history from storage, and block-level markers from earlier turns are not carried over. Send only the new user message and set cache_control at the top level of the request. Each turn then reads the whole previous conversation and writes only the new message.

{
  "model": "anthropic/claude-sonnet-5",
  "previous_response_id": "resp_...",
  "cache_control": { "type": "ephemeral" },
  "input": [
    { "role": "user", "content": "And what about clause 9?" }
  ]
}

Do not resend the system prompt on a previous_response_id turn. The stored history already contains it, and a second copy fails validation on Anthropic models.

Avoid the top-level parameter on independent requests. It marks the last block, which is the user message, so requests with different questions never share a prefix and each one pays the cache write premium without a read.

Example

curl -X POST https://my.orq.ai/v3/router/responses \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-sonnet-5",
    "input": [
      {
        "role": "system",
        "content": [
          {
            "type": "input_text",
            "text": "You are a senior legal assistant. The following is our complete contract template library...",
            "cache_control": { "type": "ephemeral" }
          }
        ]
      },
      {
        "role": "user",
        "content": "Summarize clause 7 of the NDA template."
      }
    ]
  }'

curl -X POST https://my.orq.ai/v3/router/chat/completions \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-sonnet-5",
    "messages": [
      {
        "role": "system",
        "content": [
          {
            "type": "text",
            "text": "You are a senior legal assistant. The following is our complete contract template library...",
            "cache_control": { "type": "ephemeral" }
          }
        ]
      },
      {
        "role": "user",
        "content": "Summarize clause 7 of the NDA template."
      }
    ]
  }'

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.ORQ_API_KEY,
  baseURL: "https://my.orq.ai/v3/router",
});

const response = await client.responses.create({
  model: "anthropic/claude-sonnet-5",
  input: [
    {
      role: "system",
      content: [
        {
          type: "input_text",
          text: "You are a senior legal assistant. The following is our complete contract template library...",
          cache_control: { type: "ephemeral" },
        },
      ],
    },
    { role: "user", content: "Summarize clause 7 of the NDA template." },
  ],
});

console.log(response.output_text);

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ.get("ORQ_API_KEY"),
    base_url="https://my.orq.ai/v3/router",
)

response = client.responses.create(
    model="anthropic/claude-sonnet-5",
    input=[
        {
            "role": "system",
            "content": [
                {
                    "type": "input_text",
                    "text": "You are a senior legal assistant. The following is our complete contract template library...",
                    "cache_control": {"type": "ephemeral"},
                }
            ],
        },
        {"role": "user", "content": "Summarize clause 7 of the NDA template."},
    ],
)

print(response.output_text)

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.ORQ_API_KEY,
  baseURL: "https://my.orq.ai/v3/router",
});

const response = await client.chat.completions.create({
  model: "anthropic/claude-sonnet-5",
  messages: [
    {
      role: "system",
      content: [
        {
          type: "text",
          text: "You are a senior legal assistant. The following is our complete contract template library...",
          cache_control: { type: "ephemeral" },
        },
      ],
    },
    { role: "user", content: "Summarize clause 7 of the NDA template." },
  ],
});

console.log(response.choices[0].message.content);

from openai import OpenAI
import os

client = OpenAI(
    api_key=os.environ.get("ORQ_API_KEY"),
    base_url="https://my.orq.ai/v3/router",
)

response = client.chat.completions.create(
    model="anthropic/claude-sonnet-5",
    messages=[
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "You are a senior legal assistant. The following is our complete contract template library...",
                    "cache_control": {"type": "ephemeral"},
                }
            ],
        },
        {"role": "user", "content": "Summarize clause 7 of the NDA template."},
    ],
)

print(response.choices[0].message.content)

OpenAI

Prompt caching on OpenAI models is fully automatic. No cache_control or any request changes are required. The AI Gateway forwards requests normally; OpenAI caches the prompt prefix on its side and applies the discount transparently. Caching activates on prompts longer than 1,024 tokens, in 128-token increments from that threshold. The API caches the longest matching prefix from prior requests on the same machine. Cache retention duration is model-dependent and determined by OpenAI. Refer to OpenAI’s prompt caching documentation for the current retention policy per model. Cache hits are reflected in the response usage object. See Usage in the response below.

OpenAI

Set up your OpenAI API key to use GPT models with automatic prompt caching.

Google Gemini

Google Gemini supports two caching modes through the AI Gateway. Implicit caching is enabled by default on Gemini 2.5 and newer models. No request changes are needed. The AI Gateway forwards requests normally and Google applies the cache discount automatically when a matching prefix exists. Implicit caching activates at a model-dependent minimum: 2,048 tokens on Gemini 2.5 models, higher on newer generations. Verify current thresholds and rates in Google’s caching documentation. Explicit caching is managed by the AI Gateway on supported Gemini models (Gemini 2.5 and newer). Add cache_control markers to system or user message parts (text and files), using the same syntax as on Anthropic models. The gateway creates a Google cache object for the marked content, reuses it on subsequent requests that mark the same content, and lets it expire after the TTL (5m by default, 1h supported). There is no need to create or reference Google cache objects directly. Explicit caching works on Vertex AI deployments with service-account auth. The gateway skips it on Google AI (Gemini API) when the request has a system prompt or tools, and entirely on Vertex AI Express (API-key) deployments, where only implicit caching applies.

Google AI

Set up your Google AI API key to use Gemini models with implicit prompt caching.

Usage in the response

Every endpoint reports cache reads and cache writes in the usage object. Field names follow the endpoint’s own response shape, so the same request looks slightly different on each API.

Responses API

On /v3/router/responses, cache activity is reported under usage.input_tokens_details. The same fields appear in the final response.completed event when streaming.

Field	Meaning
`cached_tokens`	Tokens read from the cache.
`cache_creation_tokens`	Tokens written to the cache, across all TTLs.
`cache_write_tokens`	Same value as `cache_creation_tokens`, kept for compatibility.
`cache_creation_5m_tokens`	Tokens written with the 5-minute TTL. Present only on a write with that TTL.
`cache_creation_1h_tokens`	Tokens written with the 1-hour TTL. Present only on a write with that TTL.

A first request on a new prefix writes the cache:

{
  "usage": {
    "input_tokens": 2673,
    "output_tokens": 30,
    "total_tokens": 2703,
    "input_tokens_details": {
      "cached_tokens": 0,
      "cache_creation_tokens": 2669,
      "cache_write_tokens": 2669,
      "cache_creation_5m_tokens": 2669
    }
  }
}

A later request on the same prefix reads it. The per-TTL fields are omitted when nothing was written:

{
  "usage": {
    "input_tokens": 2673,
    "output_tokens": 30,
    "total_tokens": 2703,
    "input_tokens_details": {
      "cached_tokens": 2669,
      "cache_creation_tokens": 0,
      "cache_write_tokens": 0
    }
  }
}

input_tokens includes cached tokens, matching the OpenAI convention.

Chat Completions API

On /v3/router/chat/completions, cache activity is reported under usage.prompt_tokens_details. The per-TTL split is not available on this endpoint.

{
  "usage": {
    "prompt_tokens": 2673,
    "completion_tokens": 30,
    "total_tokens": 2703,
    "prompt_tokens_details": {
      "cached_tokens": 2669,
      "cache_creation_tokens": 0,
      "audio_tokens": 0
    }
  }
}

Anthropic Messages API

On /v3/anthropic/v1/messages, usage uses the native Anthropic field names. input_tokens counts only uncached tokens. Cache fields are present only when non-zero: a write omits cache_read_input_tokens, and a pure read omits cache_creation_input_tokens and the nested cache_creation object.

{
  "usage": {
    "input_tokens": 4,
    "output_tokens": 30,
    "cache_creation_input_tokens": 2669,
    "cache_creation": {
      "ephemeral_5m_input_tokens": 2669,
      "ephemeral_1h_input_tokens": 0
    }
  }
}

Cache usage in Traces

Cache reads and writes are recorded on every LLM span, so they can be inspected per request in Traces and aggregated in Analytics.

Where	What is shown
Span attributes	`gen_ai.usage.prompt_tokens_details.cached_tokens`, `gen_ai.usage.prompt_tokens_details.cache_creation_tokens`, and the per-TTL `gen_ai.usage.prompt_tokens_details.cache_creation_5m_tokens` / `gen_ai.usage.prompt_tokens_details.cache_creation_1h_tokens`.
Economics panel and trace list	Cached input tokens, plus `orq.billing.cache_read_cost` and `orq.billing.cache_write_cost` in USD. The write cost already includes the 1.25x or 2x TTL premium.
Analytics	Cached tokens and cache read/write cost roll up into the workspace token and cost breakdowns.

The per-TTL token split is stored as a span attribute only. It is not surfaced as a separate column in the trace list or in Analytics. See Span attributes for the token usage and billing attributes and Token and cost tracking for how cache costs are computed.

Overview

APIs

MCP Portal

Routing

Security & Privacy

Model Capabilities

Server Tools

Common Scenarios

Prompt caching for reduced token costs

Overview

Anthropic

Supported models

Enabling caching

Minimum token thresholds

Cache TTL

Multi-turn conversations

Example

OpenAI

OpenAI

Google Gemini

Google AI

Usage in the response

Responses API

Chat Completions API

Anthropic Messages API

Cache usage in Traces

​Overview

​Anthropic

​Supported models

​Enabling caching

​Minimum token thresholds

​Cache TTL

​Multi-turn conversations

​Example

​OpenAI

OpenAI

​Google Gemini

Google AI

​Usage in the response

​Responses API

​Chat Completions API

​Anthropic Messages API

​Cache usage in Traces

Overview

Anthropic

Supported models

Enabling caching

Minimum token thresholds

Cache TTL

Multi-turn conversations

Example

OpenAI

Google Gemini

Usage in the response

Responses API

Chat Completions API

Anthropic Messages API

Cache usage in Traces