Batches

Batches are the best way to process a large volume of requests: they run without the throughput limits of real-time requests, at a fraction of the price. The Batches API runs large offline workloads without a request per line. Upload a JSONL file of requests, create a batch against a supported endpoint, and download the results once the batch reaches a terminal state. The API is OpenAI-compatible at /v3/router/batches. Batch requests, response objects, status values, and result files follow the OpenAI batch format, with Orq.ai additions for the endpoint list, listing filters, and project scoping.

Provider batches run against models that declare batch support for the endpoint and are priced at the provider’s batch rates. Check batch support for a model on the Models page.

Supported endpoints

Endpoint	Runs through	Providers
`/v1/responses`	Provider batch API	OpenAI, Google, Anthropic, AWS Bedrock (Anthropic models)
`/v1/chat/completions`	Provider batch API	OpenAI, Google, Anthropic, AWS Bedrock (Anthropic models), Mistral
`/v1/embeddings`	Provider batch API	OpenAI, Mistral
`/v1/completions`	Provider batch API	OpenAI
`/v1/moderations`	Provider batch API	OpenAI, Mistral
`/v3/router/decisions`	Orq.ai workers	Any model enabled for Decisions

Provider batches run through the provider’s batch service. Decisions batches run on Orq.ai workers, which pace the lines and check budgets per line. The OpenAI SDK type definitions declare /v1/responses, /v1/chat/completions, /v1/embeddings, and /v1/completions. Send /v1/moderations and /v3/router/decisions with the HTTP API.

Run a batch

Create a batch from the console or the API.

AI Studio
API & SDK

Open Batches under AI Gateway. The list shows one row per batch with its ID, status, model, endpoint, request counts, and creation time.
Click Batch.
Choose the JSONL file, pick the endpoint, add metadata when needed, and click Create batch. The completion window is 24h and the field is not editable.
Follow the request counts, files, and status timeline in the list or the batch detail drawer. Filter by Status and Endpoint, or search by batch ID, model, or endpoint.
Download the output or errors file from the detail drawer, or cancel a running batch from the row actions or the detail drawer.

Create batch panel with Input file, Endpoint set to /v1/responses, Completion window at 24h, and a Metadata add control. — Create a batch from the Batches screen.

Batch details drawer for a completed batch showing batch ID, model, endpoint, completion window, created and expiry times, request progress at 100 percent, the output file, status timeline, token usage, and the Download results action. — Batch details drawer with request progress, files, and the status timeline.

Write the input file

Write a JSONL file with one request per line. Each line carries a unique custom_id, "method": "POST", a url that matches the batch endpoint, and the request as body.

JSON

{"custom_id": "ticket-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openai/gpt-4o", "messages": [{"role": "user", "content": "Summarize this support ticket in one sentence."}]}}
{"custom_id": "ticket-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openai/gpt-4o", "messages": [{"role": "user", "content": "Draft a reply to this support ticket."}]}}

Upload the input file

Upload the file with the Files API and purpose set to batch. The returned file ID is the input_file_id for the batch.

curl -X POST https://my.orq.ai/v2/files \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -F "file=@requests.jsonl" \
  -F "purpose=batch"

import { Orq } from "@orq-ai/node";
import fs from "node:fs";

const orq = new Orq({
  apiKey: process.env.ORQ_API_KEY ?? "",
});

const uploaded = await orq.files.create({
  filename: "requests.jsonl",
  content: fs.readFileSync("requests.jsonl").toString("base64"),
  contentType: "application/jsonl",
  purpose: "FILE_PURPOSE_BATCH",
});

console.log(uploaded.file.fileId);

import base64
import os
from orq_ai_sdk import Orq

with open("requests.jsonl", "rb") as f:
    content = base64.b64encode(f.read()).decode("utf-8")

with Orq(api_key=os.environ["ORQ_API_KEY"]) as orq:
    uploaded = orq.files.create(
        filename="requests.jsonl",
        content=content,
        content_type="application/jsonl",
        purpose="FILE_PURPOSE_BATCH",
    )

print(uploaded.file.file_id)

Create the batch

Send the file ID, the endpoint the lines target, and a completion_window of 24h. Other values are rejected with 400.

curl -X POST https://my.orq.ai/v3/router/batches \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "file_01HZXW2K7Y8Q9M0N1P2R3S4T5V",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h"
  }'

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.ORQ_API_KEY,
  baseURL: "https://my.orq.ai/v3/router",
});

const batch = await client.batches.create({
  input_file_id: "file_01HZXW2K7Y8Q9M0N1P2R3S4T5V",
  endpoint: "/v1/chat/completions",
  completion_window: "24h",
});

console.log(batch.id);

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ORQ_API_KEY"],
    base_url="https://my.orq.ai/v3/router",
)

batch = client.batches.create(
    input_file_id="file_01HZXW2K7Y8Q9M0N1P2R3S4T5V",
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

print(batch.id)

The creation is validated up front. Every line is checked for a valid custom_id, method, url, and model before the batch is created. An invalid line rejects the file with 400 and names the line.

Poll the batch

Call GET /v3/router/batches/{batch_id} until status is completed, failed, cancelled, or expired. request_counts reports how many lines completed and failed so far.

curl https://my.orq.ai/v3/router/batches/$BATCH_ID \
  -H "Authorization: Bearer $ORQ_API_KEY"

import OpenAI from "openai";
import { setTimeout as sleep } from "node:timers/promises";

const client = new OpenAI({
  apiKey: process.env.ORQ_API_KEY,
  baseURL: "https://my.orq.ai/v3/router",
});

let batch = await client.batches.retrieve("batch_01K9X2W7Y8Q9M0N1P2R3S4T5V");

while (!["completed", "failed", "cancelled", "expired"].includes(batch.status)) {
  await sleep(5000);
  batch = await client.batches.retrieve(batch.id);
}

console.log(batch.status, batch.request_counts);

import os
import time
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ORQ_API_KEY"],
    base_url="https://my.orq.ai/v3/router",
)

batch = client.batches.retrieve("batch_01K9X2W7Y8Q9M0N1P2R3S4T5V")

while batch.status not in ("completed", "failed", "cancelled", "expired"):
    time.sleep(5)
    batch = client.batches.retrieve(batch.id)

print(batch.status, batch.request_counts)

Download the results

Call GET /v3/router/batches/{batch_id}/results to download the output file. Add ?file=errors to download the failed and unexecuted lines instead.

cURL

curl https://my.orq.ai/v3/router/batches/$BATCH_ID/results \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -o results.jsonl

The download is available once the batch reaches a terminal state. Requesting it earlier returns 409 invalid_request.

Input file

Each line follows the OpenAI batch input format.

Field	Type	Description
`custom_id`	string	Required. Unique identifier for the line, echoed in the results.
`method`	string	Required. `POST`.
`url`	string	Required. Must match the batch `endpoint`.
`body`	object	Required. The request body, including `model`.

Limits apply to the whole file and to each line.

Limit	Value
File size	100 MB
Lines per batch	99,999
Line size	20 MB
Completion window	`24h`

All lines in a provider batch must resolve to the same provider, and each model must declare batch support for the endpoint. Lines can use different models only when they share a provider and support the endpoint.

Batch statuses

Status	Meaning
`validating`	The batch is being checked before work starts.
`in_progress`	Lines are running.
`finalizing`	All lines finished and the result files are being written.
`completed`	The batch finished. Results are available.
`failed`	The batch stopped with an error. Details are in `errors`.
`expired`	The 24-hour completion window passed before all lines ran.
`cancelling`	A cancel request is being applied.
`cancelled`	The batch stopped. Lines that had not run appear in the errors file.

List and filter batches

GET /v3/router/batches returns batches in the caller’s projects, newest first.

Parameter	Type	Description
`limit`	integer	Lines per page, 1 to 100. Defaults to 20.
`after`	string	Return batches created before this batch ID.
`before`	string	Return batches created after this batch ID. Cannot be combined with `after`.
`status`	string	Comma-separated statuses, for example `completed,failed`.
`endpoint`	string	Comma-separated endpoints.
`search`	string	Case-insensitive match on batch ID, model, or endpoint.

curl "https://my.orq.ai/v3/router/batches?limit=20" \
  -H "Authorization: Bearer $ORQ_API_KEY"

import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.ORQ_API_KEY,
  baseURL: "https://my.orq.ai/v3/router",
});

const page = await client.batches.list({ limit: 20 });
console.log(page.data.length, page.has_more);

import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ORQ_API_KEY"],
    base_url="https://my.orq.ai/v3/router",
)

page = client.batches.list(limit=20)
print(len(page.data), page.has_more)

The OpenAI SDK list method sends limit and after. Send before, status, endpoint, and search over the HTTP API:

cURL

curl "https://my.orq.ai/v3/router/batches?limit=20&status=completed&endpoint=/v1/chat/completions&search=gpt-4o" \
  -H "Authorization: Bearer $ORQ_API_KEY"

The response uses the OpenAI cursor list envelope:

JSON

{
  "object": "list",
  "data": [
    {
      "id": "batch_01K9X2W7Y8Q9M0N1P2R3S4T5V",
      "object": "batch",
      "endpoint": "/v1/chat/completions",
      "input_file_id": "file_01HZXW2K7Y8Q9M0N1P2R3S4T5V",
      "completion_window": "24h",
      "status": "completed",
      "output_file_id": "file_01K9X2W7Y8Q9M0N1P2R3S4T5W",
      "created_at": 1761000000,
      "request_counts": { "total": 2, "completed": 2, "failed": 0 },
      "model": "openai/gpt-4o"
    }
  ],
  "first_id": "batch_01K9X2W7Y8Q9M0N1P2R3S4T5V",
  "last_id": "batch_01K9X2W7Y8Q9M0N1P2R3S4T5V",
  "has_more": false
}

Results

Each result line follows the OpenAI batch output format. response.body is the endpoint response for that line, and response.request_id is the trace ID of the line.

JSON

{"id": "batch_req_01K9X2W7Y8Q9M0N1P2R3S4T5V", "custom_id": "ticket-1", "response": {"status_code": 200, "request_id": "4f2294c4bd55372a354be496871aa917", "body": {"id": "chatcmpl-01K9X2W7Y8Q9M0N1P2R3S4T5W", "model": "openai/gpt-4o", "choices": [{"index": 0, "message": {"role": "assistant", "content": "The customer was charged twice for the same order."}, "finish_reason": "stop"}]}}, "error": null}

Lines that failed carry an error object instead of a response:

JSON

{"id": "batch_req_01K9X2W7Y8Q9M0N1P2R3S4T5W", "custom_id": "ticket-2", "response": null, "error": {"code": "batch_expired", "message": "The line did not run before the completion window expired."}}

Code	Meaning
`batch_cancelled`	The batch was cancelled before the line ran.
`batch_expired`	The line did not run within the 24-hour completion window.

Provider batches can also return provider-specific error codes for lines the provider rejected.

Behaviour

Validation up front: every line is checked before the batch is created, including the model and its access checks. An invalid line rejects the whole file with 400 and names the line.
Tracing and billing: a provider batch records one trace with the batch’s aggregate usage when it reaches a terminal state, priced at the provider’s batch rates. A Decisions batch runs on Orq.ai workers, and each line is traced and billed like a single request.
Budgets: Decisions batches check budgets per line. A line that hits a cost or token budget fails with 429 in its result; a requests-per-minute budget delays the line for up to two minutes before failing it.
Project scoping: a batch belongs to the project of its input file. List, retrieve, download, and cancel only reach batches in the caller’s projects.
Permissions: the API key needs the batches permission. A Decisions batch also needs the classify permission.
Rate limits: creating a batch counts as one request against plan rate limits. See Rate limits and quotas.
Cancel: POST /v3/router/batches/{batch_id}/cancel stops the lines that have not run. Calling it on a terminal batch returns the stored batch unchanged.

Files API

Upload the JSONL input file with the batch purpose.

Decisions

Batch classification requests against a Decisions model.

Background Responses

Run a single long-running response without batching.

Overview

APIs

MCP Portal

Routing

Security & Privacy

Model Capabilities

Server Tools

Common Scenarios

Supported endpoints

Run a batch

Input file

Batch statuses

List and filter batches

Results

Behaviour

Files API

Decisions

Background Responses

​Supported endpoints

​Run a batch

​Input file

​Batch statuses

​List and filter batches

​Results

​Behaviour

​Related

Files API

Decisions

Background Responses

Supported endpoints

Run a batch

Input file

Batch statuses

List and filter batches

Results

Behaviour

Related