> ## Documentation Index
> Fetch the complete documentation index at: https://docs.orq.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Build Datasets
> Create datasets to test LLM models at scale. Define inputs, messages, and expected outputs for experiments. Manage datasets via the AI Studio, API, or Orq MCP.
Datasets hold the test data that powers [Experiments](/docs/experiments/build). Each dataset row contains up to three fields:
* **Inputs**: Variables injected into the prompt at runtime, e.g. `{{firstname}}`.
* **Messages**: The prompt template, structured with system, user, and assistant roles.
* **Expected Outputs**: Reference responses evaluators compare against model outputs.
You don't need all three fields in every dataset. A dataset with only inputs, or only messages, is valid.
## Use Cases
Run the same dataset through your prompts before and after a change to verify that updates haven't degraded performance in any area.
Use the same dataset across multiple models or prompt configurations in an [Experiment](/docs/experiments/build) to find the best combination of quality, cost, and latency.
Have domain experts review and correct model outputs, then save those verified input/output pairs as a curated dataset to use as fine-tuning reference data.
Use the Orq MCP to generate hundreds of realistic test cases programmatically and add them directly to a dataset without leaving your IDE.
Build datasets with image messages for testing vision models. Supports JPEG, PNG, GIF, and WebP via the AI Studio or API.
## Create a Dataset
Use the `+` button on a [Project](/docs/projects/overview) folder and select **Dataset**. Enter a title to open the **Table View**.
The table has three columns: **Inputs**, **Messages**, and **Expected Outputs**. Add as many rows as needed.
Use the [Create Dataset API](/reference/datasets/create-a-dataset). A unique `display_name` and a `path` (the Project folder) are required.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--url https://api.orq.ai/v2/datasets \
--header 'accept: application/json' \
--header 'authorization: Bearer ORQ_API_KEY' \
--header 'content-type: application/json' \
--data '{
"display_name": "MyDataset",
"path": "Default"
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from orq_ai_sdk import Orq
import os
with Orq(api_key=os.getenv("ORQ_API_KEY", "")) as orq:
res = orq.datasets.create(request={
"display_name": "MyDataset",
"path": "Default",
})
print(res)
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { Orq } from "@orq-ai/node";
const orq = new Orq({ apiKey: process.env["ORQ_API_KEY"] ?? "" });
const result = await orq.datasets.create({
displayName: "MyDataset",
path: "Default",
});
console.log(result);
```
The response includes a `_id` field (the dataset ID) used in subsequent calls.
See the full [Create Dataset API reference](/reference/datasets/create-a-dataset).
**Create a dataset:**
```prompt wrap theme={"theme":{"light":"github-light","dark":"github-dark"}}
Create a dataset called "Support Training Data" in the Default project
```
The assistant uses `create_dataset` with the display name and path.
***
**Generate a synthetic dataset:**
```prompt wrap theme={"theme":{"light":"github-light","dark":"github-dark"}}
Generate 50 realistic customer support questions about a SaaS product and create a dataset called "Support Training Data"
```
The assistant generates the entries, uses `create_dataset` to create the dataset, then uses `create_datapoints` to add all entries in bulk.
***
**Find an existing dataset:**
```prompt wrap theme={"theme":{"light":"github-light","dark":"github-dark"}}
Find the "user-queries" dataset in my workspace
```
The assistant uses `search_entities` with `type: "dataset"` to locate it by name.
### Add Datapoints
**Manually:** Click **Add Row** and fill in each cell.
**From CSV:** Click **Import** and drag-and-drop a `.csv` file. Map each CSV column to a Dataset field (Inputs, Messages, Expected Outputs). Each row becomes a separate datapoint.
Use the [Create Datapoints API](/reference/datasets/create-a-datapoint). Send between 1 and 5,000 datapoints per request. Requests over 500 datapoints are automatically chunked.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request POST \
--url https://api.orq.ai/v2/datasets/DATASET_ID/datapoints \
--header 'accept: application/json' \
--header 'authorization: Bearer ORQ_API_KEY' \
--header 'content-type: application/json' \
--data '[
{
"inputs": {"country": "France"},
"messages": [
{"role": "user", "content": "Capital of {{country}}?"},
{"role": "assistant", "content": "Paris"}
],
"expected_output": "Paris"
},
{
"inputs": {"country": "Germany"},
"messages": [
{"role": "user", "content": "Capital of {{country}}?"},
{"role": "assistant", "content": "Berlin"}
],
"expected_output": "Berlin"
}
]'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from orq_ai_sdk import Orq
import os
with Orq(api_key=os.getenv("ORQ_API_KEY", "")) as orq:
res = orq.datasets.create_datapoint(
dataset_id="DATASET_ID",
request_body=[
{
"inputs": {"country": "France"},
"messages": [
{"role": "user", "content": "Capital of {{country}}?"},
{"role": "assistant", "content": "Paris"}
],
"expected_output": "Paris"
},
{
"inputs": {"country": "Germany"},
"messages": [
{"role": "user", "content": "Capital of {{country}}?"},
{"role": "assistant", "content": "Berlin"}
],
"expected_output": "Berlin"
}
]
)
print(f"Created {len(res)} datapoints")
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { Orq } from "@orq-ai/node";
const orq = new Orq({ apiKey: process.env["ORQ_API_KEY"] ?? "" });
await orq.datasets.createDatapoint({
datasetId: "DATASET_ID",
requestBody: [
{
inputs: { country: "France" },
messages: [
{ role: "user", content: "Capital of {{country}}?" },
{ role: "assistant", content: "Paris" }
],
expectedOutput: "Paris"
},
{
inputs: { country: "Germany" },
messages: [
{ role: "user", content: "Capital of {{country}}?" },
{ role: "assistant", content: "Berlin" }
],
expectedOutput: "Berlin"
}
]
});
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from orq_ai_sdk import Orq
import os
with Orq(api_key=os.getenv("ORQ_API_KEY", "")) as orq:
datapoints = []
for i in range(1000):
datapoints.append({
"inputs": {"number": i, "operation": "square"},
"messages": [
{"role": "user", "content": f"What is {i} squared?"},
{"role": "assistant", "content": f"{i} squared is {i**2}"}
],
"expected_output": str(i**2)
})
res = orq.datasets.create_datapoint(
dataset_id="DATASET_ID",
request_body=datapoints
)
print(f"Created {len(res)} datapoints")
```
See the full [Create Datapoints API reference](/reference/datasets/create-a-datapoint).
**Add datapoints to an existing dataset:**
```prompt wrap theme={"theme":{"light":"github-light","dark":"github-dark"}}
Import this JSON array as datapoints into the "Support Training Data" dataset
```
The assistant uses `create_datapoints` to add entries in batches (max 100 per call).
***
**Update a datapoint:**
```prompt wrap theme={"theme":{"light":"github-light","dark":"github-dark"}}
Update the expected output of datapoint ID "abc123" in the "user-queries" dataset to "New expected answer"
```
The assistant uses `update_datapoint` with the datapoint ID and updated fields.
***
**Clean up a dataset:**
```prompt wrap theme={"theme":{"light":"github-light","dark":"github-dark"}}
Delete all datapoints in the "staging-tests" dataset that have an empty expected_output field
```
The assistant uses `list_datapoints` to retrieve all entries, filters for empty `expected_output`, then uses `delete_datapoints` to remove them in batches.
### Create Image Datasets
Start by [creating a dataset](#create-a-dataset), then add messages with images.
When editing a message cell:
1. Click **Add image** in the message editor.
2. Choose how to provide the image:
* **Upload locally**: Select a file from your computer.
* **Enter URL**: Paste an image URL directly.
**Supported formats:** JPEG, PNG, GIF, WebP.
Images must be encoded as base64 data URLs before adding to the dataset.
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl -X POST https://api.orq.ai/v2/datasets \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"display_name": "Image Analysis Dataset",
"path": "Default"
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
response = orq.datasets.create(
request={"display_name": "Image Analysis Dataset", "path": "Default"}
)
dataset_id = response.id
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const dataset = await orq.datasets.create({
displayName: "Image Analysis Dataset",
path: "Default",
});
const datasetId = dataset.id;
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
import base64
def image_to_base64(image_path):
with open(image_path, "rb") as f:
encoded = base64.b64encode(f.read()).decode("utf-8")
ext = image_path.lower().split(".")[-1]
mime_types = {"png": "image/png", "gif": "image/gif", "webp": "image/webp",
"jpg": "image/jpeg", "jpeg": "image/jpeg"}
mime_type = mime_types.get(ext, "image/jpeg")
return f"data:{mime_type};base64,{encoded}"
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import fs from "fs";
import path from "path";
function imageToBase64(imagePath: string): string {
const encoded = fs.readFileSync(imagePath).toString("base64");
const ext = path.extname(imagePath).toLowerCase();
const mimeTypes: Record = {
".png": "image/png", ".gif": "image/gif", ".webp": "image/webp",
".jpg": "image/jpeg", ".jpeg": "image/jpeg",
};
return `data:${mimeTypes[ext] || "image/jpeg"};base64,${encoded}`;
}
```
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
DATASET_ID="DATASET_ID"
IMAGE_DATA=$(base64 < "/path/to/image.jpg" | tr -d '\n')
curl -X POST "https://api.orq.ai/v2/datasets/$DATASET_ID/datapoints" \
-H "Authorization: Bearer $ORQ_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "Describe what you see in this image"},
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,'$IMAGE_DATA'", "detail": "auto"}}
]
}]
}'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
orq.datasets.create_datapoint(
dataset_id=dataset_id,
request_body=[{
"messages": [{
"role": "user",
"content": [
{"type": "text", "text": "Describe what you see in this image"},
{"type": "image_url", "image_url": {"url": image_to_base64("/path/to/image.jpg"), "detail": "auto"}}
]
}]
}]
)
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
await orq.datasets.createDatapoint({
datasetId,
requestBody: [{
messages: [{
role: "user",
content: [
{ type: "text", text: "Describe what you see in this image" },
{ type: "image_url", imageUrl: { url: imageToBase64("/path/to/image.jpg"), detail: "auto" } }
]
}]
}]
});
```
**Detail parameter:**
| Value | Behaviour |
| -------------------- | ------------------------------------------- |
| `auto` (recommended) | Automatically optimises based on image size |
| `low` | Faster processing, lower token usage |
| `high` | More detailed analysis, higher token usage |
**Supported formats:**
| Format | MIME Type | Extension |
| ------ | ------------ | --------------- |
| JPEG | `image/jpeg` | `.jpg`, `.jpeg` |
| PNG | `image/png` | `.png` |
| GIF | `image/gif` | `.gif` |
| WebP | `image/webp` | `.webp` |
**Common errors:**
| Error | Cause | Solution |
| ------------------ | ----------------------- | -------------------------------------------- |
| Invalid API key | Authentication failed | Check your API key in the workspace settings |
| File not found | Image path is incorrect | Verify the path and file permissions |
| Unsupported format | Format not supported | Convert to JPEG, PNG, GIF, or WebP |
| Payload too large | Image file is too large | Compress or resize before upload |
### Create Curated Datasets
Curated datasets are human-evaluated input and output sets: a prompt paired with a verified expected output. They are used for fine-tuning and as a gold-standard reference in Experiments.
Within any module, open the **Logs** tab and select a log entry. The **Feedback** panel appears on the right.
To add a correction, click **Add correction** below the assistant response:
Edit the response in the **Correction** message that opens, then click **Save**.
Click the **Add to Dataset** icon at the top-right of the response to save the corrected entry to a dataset:
Import a curated dataset into an [Experiment](/docs/experiments/build), attach an [Evaluator](/docs/evaluators/overview), and see which model or prompt scores best against the curated reference outputs.
## List and Retrieve Datasets
**List datasets:**
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request GET \
--url https://api.orq.ai/v2/datasets \
--header 'accept: application/json' \
--header 'authorization: Bearer ORQ_API_KEY'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
from orq_ai_sdk import Orq
import os
with Orq(api_key=os.getenv("ORQ_API_KEY", "")) as orq:
res = orq.datasets.list(limit=10)
print(res)
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
import { Orq } from "@orq-ai/node";
const orq = new Orq({ apiKey: process.env["ORQ_API_KEY"] ?? "" });
const result = await orq.datasets.list({});
console.log(result);
```
**Retrieve a dataset by ID:**
```bash cURL theme={"theme":{"light":"github-light","dark":"github-dark"}}
curl --request GET \
--url https://api.orq.ai/v2/datasets/DATASET_ID \
--header 'accept: application/json' \
--header 'authorization: Bearer ORQ_API_KEY'
```
```python Python theme={"theme":{"light":"github-light","dark":"github-dark"}}
with Orq(api_key=os.getenv("ORQ_API_KEY", "")) as orq:
res = orq.datasets.retrieve(dataset_id="DATASET_ID")
print(res)
```
```typescript TypeScript theme={"theme":{"light":"github-light","dark":"github-dark"}}
const result = await orq.datasets.retrieve({ datasetId: "DATASET_ID" });
console.log(result);
```
See the [List Datasets](/reference/datasets/list-datasets) and [Retrieve a Dataset](/reference/datasets/retrieve-a-dataset) API references.
**List datasets in a project:**
```prompt wrap theme={"theme":{"light":"github-light","dark":"github-dark"}}
List all datasets in my workspace
```
The assistant uses `search_entities` with `type: "dataset"`.
***
**Find a specific dataset:**
```prompt wrap theme={"theme":{"light":"github-light","dark":"github-dark"}}
Find the "user-queries" dataset and show me its datapoints
```
The assistant uses `search_entities` to locate the dataset, then `list_datapoints` to retrieve its entries.