> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orq.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Batches

> Submit large offline workloads as JSONL batches through the /v3/router/batches API and download OpenAI-format results.

Batches are the best way to process a large volume of requests: they run without the throughput limits of real-time requests, at a fraction of the price.

The Batches API runs large offline workloads without a request per line. Upload a JSONL file of requests, create a batch against a supported endpoint, and download the results once the batch reaches a terminal state.

The API is OpenAI-compatible at `/v3/router/batches`. Batch requests, response objects, status values, and result files follow the OpenAI batch format, with **Orq.ai** additions for the endpoint list, listing filters, and project scoping.

> [!NOTE]
>
> Provider batches run against models that declare batch support for the endpoint and are priced at the provider's batch rates. Check batch support for a model on the [Models page](/ai-gateway/using-the-router).

## Supported endpoints

| Endpoint | Runs through | Providers |
| - | - | - |
| `/v1/responses` | Provider batch API | OpenAI, Google, Anthropic, AWS Bedrock (Anthropic models) |
| `/v1/chat/completions` | Provider batch API | OpenAI, Google, Anthropic, AWS Bedrock (Anthropic models), Mistral |
| `/v1/embeddings` | Provider batch API | OpenAI, Mistral |
| `/v1/completions` | Provider batch API | OpenAI |
| `/v1/moderations` | Provider batch API | OpenAI, Mistral |
| `/v3/router/decisions` | **Orq.ai** workers | Any model enabled for [Decisions](/ai-gateway/features/decisions) |

Provider batches run through the provider's batch service. Decisions batches run on **Orq.ai** workers, which pace the lines and check budgets per line.

The OpenAI SDK type definitions declare `/v1/responses`, `/v1/chat/completions`, `/v1/embeddings`, and `/v1/completions`. Send `/v1/moderations` and `/v3/router/decisions` with the HTTP API.

## Run a batch

Create a batch from the console or the API.

**AI Studio**

1. Open **Batches** under **AI Gateway**. The list shows one row per batch with its ID, status, model, endpoint, request counts, and creation time.
2. Click <kbd> Batch</kbd>.
3. Choose the JSONL file, pick the endpoint, add metadata when needed, and click **Create batch**. The completion window is `24h` and the field is not editable.
4. Follow the request counts, files, and status timeline in the list or the batch detail drawer. Filter by **Status** and **Endpoint**, or search by batch ID, model, or endpoint.
5. Download the output or errors file from the detail drawer, or cancel a running batch from the row actions or the detail drawer.

![Create batch panel with Input file, Endpoint set to /v1/responses, Completion window at 24h, and a Metadata add control.](https://mintcdn.com/orqai/xDmNHFU8LGAKgsSa/images/batches-create-panel.png?fit=max\&auto=format\&n=xDmNHFU8LGAKgsSa\&q=85\&s=69709cad0980bf60b7353a70808ac4dc)

![Batch details drawer for a completed batch showing batch ID, model, endpoint, completion window, created and expiry times, request progress at 100 percent, the output file, status timeline, token usage, and the Download results action.](https://mintcdn.com/orqai/xDmNHFU8LGAKgsSa/images/batches-detail-drawer.png?fit=max\&auto=format\&n=xDmNHFU8LGAKgsSa\&q=85\&s=430f588986d0e31c49b1bd920f83be48)

**API & SDK**

**1. Write the input file**

Write a JSONL file with one request per line. Each line carries a unique `custom_id`, `"method": "POST"`, a `url` that matches the batch `endpoint`, and the request as `body`.

```json JSON theme={"theme":{"light":"github-light","dark":"github-dark"}}
{"custom_id": "ticket-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openai/gpt-4o", "messages": [{"role": "user", "content": "Summarize this support ticket in one sentence."}]}}
{"custom_id": "ticket-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "openai/gpt-4o", "messages": [{"role": "user", "content": "Draft a reply to this support ticket."}]}}
```

**2. Upload the input file**

Upload the file with the [Files API](/ai-studio/ai-engineering/files) and `purpose` set to `batch`. The returned file ID is the `input_file_id` for the batch.

```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://my.orq.ai/v2/files \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -F "file=@requests.jsonl" \
  -F "purpose=batch"
```

```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { Orq } from "@orq-ai/node";
import fs from "node:fs";

const orq = new Orq({
  apiKey: process.env.ORQ_API_KEY ?? "",
});

const uploaded = await orq.files.create({
  filename: "requests.jsonl",
  content: fs.readFileSync("requests.jsonl").toString("base64"),
  contentType: "application/jsonl",
  purpose: "FILE_PURPOSE_BATCH",
});

console.log(uploaded.file.fileId);
```

```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import base64
import os
from orq_ai_sdk import Orq

with open("requests.jsonl", "rb") as f:
    content = base64.b64encode(f.read()).decode("utf-8")

with Orq(api_key=os.environ["ORQ_API_KEY"]) as orq:
    uploaded = orq.files.create(
        filename="requests.jsonl",
        content=content,
        content_type="application/jsonl",
        purpose="FILE_PURPOSE_BATCH",
    )

print(uploaded.file.file_id)
```

**3. Create the batch**

Send the file ID, the endpoint the lines target, and a `completion_window` of `24h`. Other values are rejected with `400`.

```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://my.orq.ai/v3/router/batches \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "input_file_id": "file_01HZXW2K7Y8Q9M0N1P2R3S4T5V",
    "endpoint": "/v1/chat/completions",
    "completion_window": "24h"
  }'
```

```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.ORQ_API_KEY,
  baseURL: "https://my.orq.ai/v3/router",
});

const batch = await client.batches.create({
  input_file_id: "file_01HZXW2K7Y8Q9M0N1P2R3S4T5V",
  endpoint: "/v1/chat/completions",
  completion_window: "24h",
});

console.log(batch.id);
```

```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ORQ_API_KEY"],
    base_url="https://my.orq.ai/v3/router",
)

batch = client.batches.create(
    input_file_id="file_01HZXW2K7Y8Q9M0N1P2R3S4T5V",
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

print(batch.id)
```

The creation is validated up front. Every line is checked for a valid `custom_id`, `method`, `url`, and model before the batch is created. An invalid line rejects the file with `400` and names the line.

**4. Poll the batch**

Call `GET /v3/router/batches/{batch_id}` until `status` is `completed`, `failed`, `cancelled`, or `expired`. `request_counts` reports how many lines completed and failed so far.

```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl https://my.orq.ai/v3/router/batches/$BATCH_ID \
  -H "Authorization: Bearer $ORQ_API_KEY"
```

```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from "openai";
import { setTimeout as sleep } from "node:timers/promises";

const client = new OpenAI({
  apiKey: process.env.ORQ_API_KEY,
  baseURL: "https://my.orq.ai/v3/router",
});

let batch = await client.batches.retrieve("batch_01K9X2W7Y8Q9M0N1P2R3S4T5V");

while (!["completed", "failed", "cancelled", "expired"].includes(batch.status)) {
  await sleep(5000);
  batch = await client.batches.retrieve(batch.id);
}

console.log(batch.status, batch.request_counts);
```

```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
import time
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ORQ_API_KEY"],
    base_url="https://my.orq.ai/v3/router",
)

batch = client.batches.retrieve("batch_01K9X2W7Y8Q9M0N1P2R3S4T5V")

while batch.status not in ("completed", "failed", "cancelled", "expired"):
    time.sleep(5)
    batch = client.batches.retrieve(batch.id)

print(batch.status, batch.request_counts)
```

**5. Download the results**

Call `GET /v3/router/batches/{batch_id}/results` to download the output file. Add `?file=errors` to download the failed and unexecuted lines instead.

```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl https://my.orq.ai/v3/router/batches/$BATCH_ID/results \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -o results.jsonl
```

The download is available once the batch reaches a terminal state. Requesting it earlier returns `409 invalid_request`.

## Input file

Each line follows the OpenAI batch input format.

| Field | Type | Description |
| - | - | - |
| `custom_id` | string | Required. Unique identifier for the line, echoed in the results. |
| `method` | string | Required. `POST`. |
| `url` | string | Required. Must match the batch `endpoint`. |
| `body` | object | Required. The request body, including `model`. |

Limits apply to the whole file and to each line.

| Limit | Value |
| - | - |
| File size | 100 MB |
| Lines per batch | 99,999 |
| Line size | 20 MB |
| Completion window | `24h` |

All lines in a provider batch must resolve to the same provider, and each model must declare batch support for the endpoint. Lines can use different models only when they share a provider and support the endpoint.

## Batch statuses

| Status | Meaning |
| - | - |
| `validating` | The batch is being checked before work starts. |
| `in_progress` | Lines are running. |
| `finalizing` | All lines finished and the result files are being written. |
| `completed` | The batch finished. Results are available. |
| `failed` | The batch stopped with an error. Details are in `errors`. |
| `expired` | The 24-hour completion window passed before all lines ran. |
| `cancelling` | A cancel request is being applied. |
| `cancelled` | The batch stopped. Lines that had not run appear in the errors file. |

## List and filter batches

`GET /v3/router/batches` returns batches in the caller's projects, newest first.

| Parameter | Type | Description |
| - | - | - |
| `limit` | integer | Lines per page, 1 to 100. Defaults to 20. |
| `after` | string | Return batches created before this batch ID. |
| `before` | string | Return batches created after this batch ID. Cannot be combined with `after`. |
| `status` | string | Comma-separated statuses, for example `completed,failed`. |
| `endpoint` | string | Comma-separated endpoints. |
| `search` | string | Case-insensitive match on batch ID, model, or endpoint. |

```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl "https://my.orq.ai/v3/router/batches?limit=20" \
  -H "Authorization: Bearer $ORQ_API_KEY"
```

```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import OpenAI from "openai";

const client = new OpenAI({
  apiKey: process.env.ORQ_API_KEY,
  baseURL: "https://my.orq.ai/v3/router",
});

const page = await client.batches.list({ limit: 20 });
console.log(page.data.length, page.has_more);
```

```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import os
from openai import OpenAI

client = OpenAI(
    api_key=os.environ["ORQ_API_KEY"],
    base_url="https://my.orq.ai/v3/router",
)

page = client.batches.list(limit=20)
print(len(page.data), page.has_more)
```

The OpenAI SDK `list` method sends `limit` and `after`. Send `before`, `status`, `endpoint`, and `search` over the HTTP API:

```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl "https://my.orq.ai/v3/router/batches?limit=20&status=completed&endpoint=/v1/chat/completions&search=gpt-4o" \
  -H "Authorization: Bearer $ORQ_API_KEY"
```

The response uses the OpenAI cursor list envelope:

```json JSON theme={"theme":{"light":"github-light","dark":"github-dark"}}
{
  "object": "list",
  "data": [
    {
      "id": "batch_01K9X2W7Y8Q9M0N1P2R3S4T5V",
      "object": "batch",
      "endpoint": "/v1/chat/completions",
      "input_file_id": "file_01HZXW2K7Y8Q9M0N1P2R3S4T5V",
      "completion_window": "24h",
      "status": "completed",
      "output_file_id": "file_01K9X2W7Y8Q9M0N1P2R3S4T5W",
      "created_at": 1761000000,
      "request_counts": { "total": 2, "completed": 2, "failed": 0 },
      "model": "openai/gpt-4o"
    }
  ],
  "first_id": "batch_01K9X2W7Y8Q9M0N1P2R3S4T5V",
  "last_id": "batch_01K9X2W7Y8Q9M0N1P2R3S4T5V",
  "has_more": false
}
```

## Results

Each result line follows the OpenAI batch output format. `response.body` is the endpoint response for that line, and `response.request_id` is the trace ID of the line.

```json JSON theme={"theme":{"light":"github-light","dark":"github-dark"}}
{"id": "batch_req_01K9X2W7Y8Q9M0N1P2R3S4T5V", "custom_id": "ticket-1", "response": {"status_code": 200, "request_id": "4f2294c4bd55372a354be496871aa917", "body": {"id": "chatcmpl-01K9X2W7Y8Q9M0N1P2R3S4T5W", "model": "openai/gpt-4o", "choices": [{"index": 0, "message": {"role": "assistant", "content": "The customer was charged twice for the same order."}, "finish_reason": "stop"}]}}, "error": null}
```

Lines that failed carry an `error` object instead of a `response`:

```json JSON theme={"theme":{"light":"github-light","dark":"github-dark"}}
{"id": "batch_req_01K9X2W7Y8Q9M0N1P2R3S4T5W", "custom_id": "ticket-2", "response": null, "error": {"code": "batch_expired", "message": "The line did not run before the completion window expired."}}
```

| Code | Meaning |
| - | - |
| `batch_cancelled` | The batch was cancelled before the line ran. |
| `batch_expired` | The line did not run within the 24-hour completion window. |

Provider batches can also return provider-specific error codes for lines the provider rejected.

## Behaviour

* **Validation up front**: every line is checked before the batch is created, including the model and its access checks. An invalid line rejects the whole file with `400` and names the line.
* **Tracing and billing**: a provider batch records one trace with the batch's aggregate usage when it reaches a terminal state, priced at the provider's batch rates. A Decisions batch runs on **Orq.ai** workers, and each line is traced and billed like a single request.
* **Budgets**: Decisions batches check budgets per line. A line that hits a cost or token budget fails with `429` in its result; a requests-per-minute budget delays the line for up to two minutes before failing it.
* **Project scoping**: a batch belongs to the project of its input file. List, retrieve, download, and cancel only reach batches in the caller's projects.
* **Permissions**: the API key needs the `batches` permission. A Decisions batch also needs the `classify` permission.
* **Rate limits**: creating a batch counts as one request against plan rate limits. See [Rate limits and quotas](/ai-gateway/features/rate-limits).
* **Cancel**: `POST /v3/router/batches/{batch_id}/cancel` stops the lines that have not run. Calling it on a terminal batch returns the stored batch unchanged.

## Related

> **[Files API](/ai-studio/ai-engineering/files)**
>
> Upload the JSONL input file with the batch purpose.

> **[Decisions](/ai-gateway/features/decisions)**
>
> Batch classification requests against a Decisions model.

> **[Background Responses](/ai-gateway/features/background-responses)**
>
> Run a single long-running response without batching.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.