Build Datasets

Datasets hold the test data that powers Experiments. Each dataset row contains up to three fields:

Inputs: Variables injected into the prompt at runtime, e.g. {{firstname}}.
Messages: The prompt template, structured with system, user, and assistant roles.
Expected Output: Reference responses evaluators compare against model outputs.

You don’t need all three fields in every dataset. A dataset with only inputs, or only messages, is valid.

Use Cases

Regression and pre-deployment testing

Run the same dataset through your prompts before and after a change to verify that updates haven’t degraded performance in any area.

Compare models and prompt variants

Use the same dataset across multiple models or prompt configurations in an Experiment to find the best combination of quality, cost, and latency.

Curated datasets for fine-tuning

Have domain experts review and correct model outputs, then save those verified input/output pairs as a curated dataset to use as fine-tuning reference data.

Synthetic data generation at scale

Use the Orq MCP to generate hundreds of realistic test cases programmatically and add them directly to a dataset without leaving your IDE.

Vision and image datasets

Build datasets with image messages for testing vision models. Supports JPEG, PNG, GIF, and WebP via the AI Studio or API.

Create a Dataset

AI Studio
API & SDK
CLI
MCP

Use the button on a Project folder and select Dataset. Enter a Name in the Create dataset dialog and select Create.The table has three columns: Inputs, Messages, and Expected Output. Add as many rows as needed. Create a Dataset

Use the Create Dataset API. A unique display_name and a path (the Project folder) are required.

curl --request POST \
     --url https://my.orq.ai/v2/datasets \
     --header 'accept: application/json' \
     --header "authorization: Bearer $ORQ_API_KEY" \
     --header 'content-type: application/json' \
     --data '{
  "display_name": "MyDataset",
  "path": "Default"
}'

from orq_ai_sdk import Orq
import os

with Orq(api_key=os.getenv("ORQ_API_KEY", "")) as orq:
    res = orq.datasets.create(
        display_name="MyDataset",
        path="Default",
    )
    print(res)

import { Orq } from "@orq-ai/node";

const orq = new Orq({ apiKey: process.env["ORQ_API_KEY"] ?? "" });

const result = await orq.datasets.create({
  displayName: "MyDataset",
  path: "Default",
});

console.log(result);

The response includes a _id field (the dataset ID) used in subsequent calls.

See the full Create Dataset API reference.

Create a dataset using orq datasets create:

orq datasets create \
  --display-name "MyDataset" \
  --path Default

See install and setup to get started with the CLI. Run orq datasets create --help for the full flag reference.

Create a dataset:

Create a dataset called "Support Training Data" in the Default project

The assistant uses create_dataset with the display name and path.

Generate a synthetic dataset:

Generate 50 realistic customer support questions about a SaaS product and create a dataset called "Support Training Data"

The assistant generates the entries, uses create_dataset to create the dataset, then uses create_datapoints to add all entries in bulk.

Find an existing dataset:

Find the "user-queries" dataset in my workspace

The assistant uses search_entities with type: "dataset" to locate it by name.

Add Datapoints

AI Studio
API & SDK
CLI
MCP

Manually: Click Add Row and fill in each cell.From CSV: Click Import and drag-and-drop a .csv file. Map each CSV column to a role: Inputs, Messages, Expected Output, or Exclude. Each row becomes a separate datapoint.

Use the Create Datapoints API. Send between 1 and 5,000 datapoints per request. Requests over 500 datapoints are automatically chunked.

curl --request POST \
     --url https://my.orq.ai/v2/datasets/DATASET_ID/datapoints \
     --header 'accept: application/json' \
     --header "authorization: Bearer $ORQ_API_KEY" \
     --header 'content-type: application/json' \
     --data '[
  {
    "inputs": {"country": "France"},
    "messages": [
      {"role": "user", "content": "Capital of {{country}}?"},
      {"role": "assistant", "content": "Paris"}
    ],
    "expected_output": "Paris"
  },
  {
    "inputs": {"country": "Germany"},
    "messages": [
      {"role": "user", "content": "Capital of {{country}}?"},
      {"role": "assistant", "content": "Berlin"}
    ],
    "expected_output": "Berlin"
  }
]'

from orq_ai_sdk import Orq
import os

with Orq(api_key=os.getenv("ORQ_API_KEY", "")) as orq:
    res = orq.datasets.create_datapoint(
        dataset_id="DATASET_ID",
        request_body=[
            {
                "inputs": {"country": "France"},
                "messages": [
                    {"role": "user", "content": "Capital of {{country}}?"},
                    {"role": "assistant", "content": "Paris"}
                ],
                "expected_output": "Paris"
            },
            {
                "inputs": {"country": "Germany"},
                "messages": [
                    {"role": "user", "content": "Capital of {{country}}?"},
                    {"role": "assistant", "content": "Berlin"}
                ],
                "expected_output": "Berlin"
            }
        ]
    )
    print(f"Created {len(res)} datapoints")

import { Orq } from "@orq-ai/node";

const orq = new Orq({ apiKey: process.env["ORQ_API_KEY"] ?? "" });

await orq.datasets.createDatapoint({
  datasetId: "DATASET_ID",
  requestBody: [
    {
      inputs: { country: "France" },
      messages: [
        { role: "user", content: "Capital of {{country}}?" },
        { role: "assistant", content: "Paris" }
      ],
      expectedOutput: "Paris"
    },
    {
      inputs: { country: "Germany" },
      messages: [
        { role: "user", content: "Capital of {{country}}?" },
        { role: "assistant", content: "Berlin" }
      ],
      expectedOutput: "Berlin"
    }
  ]
});

Large batch example (1,000 datapoints)

Python

from orq_ai_sdk import Orq
import os

with Orq(api_key=os.getenv("ORQ_API_KEY", "")) as orq:
    datapoints = []
    for i in range(1000):
        datapoints.append({
            "inputs": {"number": i, "operation": "square"},
            "messages": [
                {"role": "user", "content": f"What is {i} squared?"},
                {"role": "assistant", "content": f"{i} squared is {i**2}"}
            ],
            "expected_output": str(i**2)
        })

    res = orq.datasets.create_datapoint(
        dataset_id="DATASET_ID",
        request_body=datapoints
    )
    print(f"Created {len(res)} datapoints")

See the full Create Datapoints API reference.

Add datapoints using orq datasets create-datapoint. The request body is an array of datapoints, piped via stdin:

echo '[
  {
    "inputs": {"country": "France"},
    "messages": [
      {"role": "user", "content": "Capital of {{country}}?"},
      {"role": "assistant", "content": "Paris"}
    ],
    "expected_output": "Paris"
  },
  {
    "inputs": {"country": "Germany"},
    "messages": [
      {"role": "user", "content": "Capital of {{country}}?"},
      {"role": "assistant", "content": "Berlin"}
    ],
    "expected_output": "Berlin"
  }
]' | orq datasets create-datapoint <dataset_id> --stdin

See install and setup to get started with the CLI. Run orq datasets create-datapoint --help for the full flag reference.

Add datapoints to an existing dataset:

Import this JSON array as datapoints into the "Support Training Data" dataset

The assistant uses create_datapoints to add entries in batches (max 100 per call).

Update a datapoint:

Update the expected output of datapoint ID "abc123" in the "user-queries" dataset to "New expected answer"

The assistant uses update_datapoint with the datapoint ID and updated fields.

Clean up a dataset:

Delete all datapoints in the "staging-tests" dataset that have an empty expected_output field

The assistant uses list_datapoints to retrieve all entries, filters for empty expected_output, then uses delete_datapoints to remove them in batches.

Create Image Datasets

AI Studio
API & SDK
CLI

Start by creating a dataset, then add messages with images.When editing a message cell:

Click Add image in the message editor.

Choose how to provide the image:
- Upload locally: Select a file from your computer.
- Enter URL: Paste an image URL directly.

URL input field for entering an image URL, with a Select image button to browse locally. — Enter an image URL or upload a local image file.

Animated walkthrough of adding an image to a dataset message by URL or local file upload. — Complete workflow: Adding images to dataset messages.

Supported formats: JPEG, PNG, GIF, WebP.

Images must be encoded as base64 data URLs before adding to the dataset.

Create a Dataset

curl -X POST https://my.orq.ai/v2/datasets \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "display_name": "Image Analysis Dataset",
    "path": "Default"
  }'

response = orq.datasets.create(
    display_name="Image Analysis Dataset", path="Default"
)
dataset_id = response.id

const dataset = await orq.datasets.create({
  displayName: "Image Analysis Dataset",
  path: "Default",
});
const datasetId = dataset.id;

Convert Images to Base64

import base64

def image_to_base64(image_path):
    with open(image_path, "rb") as f:
        encoded = base64.b64encode(f.read()).decode("utf-8")
    ext = image_path.lower().split(".")[-1]
    mime_types = {"png": "image/png", "gif": "image/gif", "webp": "image/webp",
                  "jpg": "image/jpeg", "jpeg": "image/jpeg"}
    mime_type = mime_types.get(ext, "image/jpeg")
    return f"data:{mime_type};base64,{encoded}"

import fs from "fs";
import path from "path";

function imageToBase64(imagePath: string): string {
  const encoded = fs.readFileSync(imagePath).toString("base64");
  const ext = path.extname(imagePath).toLowerCase();
  const mimeTypes: Record<string, string> = {
    ".png": "image/png", ".gif": "image/gif", ".webp": "image/webp",
    ".jpg": "image/jpeg", ".jpeg": "image/jpeg",
  };
  return `data:${mimeTypes[ext] || "image/jpeg"};base64,${encoded}`;
}

Add Images to the Dataset

DATASET_ID="DATASET_ID"
IMAGE_DATA=$(base64 < "/path/to/image.jpg" | tr -d '\n')

curl -X POST "https://my.orq.ai/v2/datasets/$DATASET_ID/datapoints" \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "Describe what you see in this image"},
        {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,'$IMAGE_DATA'", "detail": "auto"}}
      ]
    }]
  }'

orq.datasets.create_datapoint(
    dataset_id=dataset_id,
    request_body=[{
        "messages": [{
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe what you see in this image"},
                {"type": "image_url", "image_url": {"url": image_to_base64("/path/to/image.jpg"), "detail": "auto"}}
            ]
        }]
    }]
)

await orq.datasets.createDatapoint({
  datasetId,
  requestBody: [{
    messages: [{
      role: "user",
      content: [
        { type: "text", text: "Describe what you see in this image" },
        { type: "image_url", imageUrl: { url: imageToBase64("/path/to/image.jpg"), detail: "auto" } }
      ]
    }]
  }]
});

Detail parameter:

Value	Behaviour
`auto` (recommended)	Automatically optimises based on image size
`low`	Faster processing, lower token usage
`high`	More detailed analysis, higher token usage

Supported formats:

Format	MIME Type	Extension
JPEG	`image/jpeg`	`.jpg`, `.jpeg`
PNG	`image/png`	`.png`
GIF	`image/gif`	`.gif`
WebP	`image/webp`	`.webp`

Common errors:

Error	Cause	Solution
Invalid API key	Authentication failed	Check your API key in the workspace settings
File not found	Image path is incorrect	Verify the path and file permissions
Unsupported format	Format not supported	Convert to JPEG, PNG, GIF, or WebP
Payload too large	Image file is too large	Compress or resize before upload

Encode the image as a base64 data URL, then add it as a datapoint using orq datasets create-datapoint:

IMAGE_DATA=$(base64 < /path/to/image.jpg | tr -d '\n')

echo '[{
  "messages": [{
    "role": "user",
    "content": [
      {"type": "text", "text": "Describe what you see in this image"},
      {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,'"$IMAGE_DATA"'", "detail": "auto"}}
    ]
  }]
}]' | orq datasets create-datapoint <dataset_id> --stdin

See install and setup to get started with the CLI. Run orq datasets create-datapoint --help for the full flag reference.

Create File and PDF Datasets

A datapoint carries a document in a file content part. Datapoints take the Chat Completions shape, not the input_file shape of the Responses API:

curl --request POST \
     --url https://my.orq.ai/v2/datasets/DATASET_ID/datapoints \
     --header 'content-type: application/json' \
     --header "authorization: Bearer $ORQ_API_KEY" \
     --data '[
  {
    "inputs": {"invoice": "invoice.pdf"},
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "text", "text": "Extract the total from this invoice"},
          {"type": "file", "file": {
            "file_data": "data:application/pdf;base64,JVBERi0xLjQK...",
            "filename": "invoice.pdf"
          }}
        ]
      }
    ],
    "expected_output": "1240.00"
  }
]'

import base64
import os
from pathlib import Path

from orq_ai_sdk import Orq

pdf = base64.b64encode(Path("invoice.pdf").read_bytes()).decode("utf-8")
part = {"type": "file", "file": {"file_data": f"data:application/pdf;base64,{pdf}", "filename": "invoice.pdf"}}

with Orq(api_key=os.environ["ORQ_API_KEY"]) as orq:
    orq.datasets.create_datapoint(
        dataset_id="DATASET_ID",
        request_body=[{
            "inputs": {"invoice": "invoice.pdf"},
            "messages": [{"role": "user", "content": [
                {"type": "text", "text": "Extract the total from this invoice"},
                part,
            ]}],
            "expected_output": "1240.00",
        }],
    )

import fs from "fs";
import { Orq } from "@orq-ai/node";

const orq = new Orq({ apiKey: process.env["ORQ_API_KEY"] ?? "" });
const pdf = fs.readFileSync("invoice.pdf").toString("base64");

await orq.datasets.createDatapoint({
  datasetId: "DATASET_ID",
  requestBody: [
    {
      inputs: { invoice: "invoice.pdf" },
      messages: [
        {
          role: "user",
          content: [
            { type: "text", text: "Extract the total from this invoice" },
            {
              type: "file",
              file: { fileData: `data:application/pdf;base64,${pdf}`, filename: "invoice.pdf" },
            },
          ],
        },
      ],
      expectedOutput: "1240.00",
    },
  ],
});

In place of file_data, the part takes a file_id from an uploaded file or a uri. The messages payload counts against a 100 MB request body, and base64 adds about a third to the file size. When the datapoint is saved, base64 media in a user message moves to object storage: the stored message keeps a reference to the object instead of the bytes. The model must accept the document: Sending files to models lists the types each model takes. An Experiment or Agent run sends the row’s messages as the request and resolves the object reference, so the model receives the document.

Create Curated Datasets

Curated datasets are human-evaluated input and output sets: a prompt paired with a verified expected output. They are used for fine-tuning and as a gold-standard reference in Experiments. Within any module, open the Logs tab and select a log entry. The Feedback panel appears on the right. To add a correction, click Add correction below the assistant response:

Edit the response in the Correction message that opens, then click Save.

Original assistant response shown in purple above a Correction box in green, with the corrected text entered and a Save button. — The original and corrected responses appear side by side. The correction is shown in green.

Click the Add to Dataset icon at the top-right of the response to save the corrected entry to a dataset:

Log view showing an Add to dataset dropdown with curated dataset selected, a Replace Inputs toggle, and an Add to dataset button. — Choose to replace the inputs used during generation (recommended).

Import a curated dataset into an Experiment, attach an Evaluator, and see which model or prompt scores best against the curated reference outputs.

List and Retrieve Datasets

API & SDK
CLI
MCP

List datasets:

curl --request GET \
     --url https://my.orq.ai/v2/datasets \
     --header 'accept: application/json' \
     --header "authorization: Bearer $ORQ_API_KEY"

from orq_ai_sdk import Orq
import os

with Orq(api_key=os.getenv("ORQ_API_KEY", "")) as orq:
    res = orq.datasets.list(limit=10)
    print(res)

import { Orq } from "@orq-ai/node";

const orq = new Orq({ apiKey: process.env["ORQ_API_KEY"] ?? "" });
const result = await orq.datasets.list({});
console.log(result);

Retrieve a dataset by ID:

curl --request GET \
     --url https://my.orq.ai/v2/datasets/DATASET_ID \
     --header 'accept: application/json' \
     --header "authorization: Bearer $ORQ_API_KEY"

with Orq(api_key=os.getenv("ORQ_API_KEY", "")) as orq:
    res = orq.datasets.retrieve(dataset_id="DATASET_ID")
    print(res)

const result = await orq.datasets.retrieve({ datasetId: "DATASET_ID" });
console.log(result);

See the List Datasets and Retrieve a Dataset API references.

List and retrieve datasets using orq datasets list and orq datasets retrieve:

# List datasets
orq datasets list --limit 10

# Retrieve a dataset
orq datasets retrieve <dataset_id>

See install and setup to get started with the CLI. Run orq datasets list --help or orq datasets retrieve --help for the full flag reference.

List datasets in a project:

List all datasets in my workspace

The assistant uses search_entities with type: "dataset".

Find a specific dataset:

Find the "user-queries" dataset and show me its datapoints

The assistant uses search_entities to locate the dataset, then list_datapoints to retrieve its entries.

Overview

Managed Agents

Context Management

Test & Evaluate

Chat

Use Cases

Create a Dataset

Add Datapoints

Create Image Datasets

Create File and PDF Datasets

Create Curated Datasets

List and Retrieve Datasets