Core - Execute a run

curl 'https://my.orq.ai/v3/experiments/<experiment_id>/runs/<run_id>/execute' \
  --header 'Authorization: Bearer $ORQ_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{
    "scope": "EXECUTE_SCOPE_FULL",
    "constraints": { "llm": { "concurrency": 5, "timeout": "300s" } }
  }'

import { Orq } from '@orq-ai/node';

const client = new Orq({ apiKey: process.env.ORQ_API_KEY });

const run = await client.experiments.executeRun({
  experimentId: '<experiment_id>',
  runId: '<run_id>',
  scope: 'EXECUTE_SCOPE_FULL',
  constraints: { llm: { concurrency: 5, timeout: '300s' } },
});

console.log(run.status);

import os
from orq_ai_sdk import Orq

client = Orq(api_key=os.environ["ORQ_API_KEY"])

run = client.experiments.execute_run(
    experiment_id="<experiment_id>",
    run_id="<run_id>",
    scope="EXECUTE_SCOPE_FULL",
    constraints={"llm": {"concurrency": 5, "timeout": "300s"}},
)

print(run.status)

{
  "run_id": "<string>",
  "experiment_id": "<string>",
  "key": "<string>",
  "status": "RUN_STATUS_UNSPECIFIED",
  "dataset_id": "<string>",
  "row_count": 123,
  "cell_status_counts": {
    "not_run": 123,
    "queued": 123,
    "running": 123,
    "completed": 123,
    "failed": 123,
    "canceled": 123
  },
  "columns": [
    {
      "column_id": "<string>",
      "key": "<string>",
      "display_name": "<string>",
      "position": 123,
      "active": true,
      "prompt": {
        "prompt_config": {},
        "messages_source": "MESSAGES_SOURCE_UNSPECIFIED",
        "engine": "<string>",
        "tools": [
          {}
        ],
        "knowledge_bases": [
          {}
        ],
        "metadata": {}
      },
      "agent": {
        "agent_key": "<string>",
        "messages_source": "MESSAGES_SOURCE_UNSPECIFIED",
        "messages": [
          {}
        ],
        "engine": "<string>",
        "parent_id": "<string>",
        "agent_tag": "<string>"
      },
      "openresponse": {},
      "evaluator": {
        "evaluator_id": "<string>",
        "evaluator_name": "<string>",
        "evaluator_type": "<string>",
        "evaluator_output_type": "<string>"
      },
      "review": {
        "human_eval_id": "<string>",
        "human_eval_name": "<string>",
        "human_eval_type": "<string>"
      },
      "dataset": {
        "column_type": "<string>"
      },
      "metric": {
        "metric_key": "<string>"
      },
      "job_output": {
        "job_name": "<string>"
      },
      "width": 123
    }
  ],
  "settings": {
    "concurrency": 123,
    "timeout": "<string>",
    "max_turns": 123,
    "hidden_column_ids": [
      "<string>"
    ]
  },
  "started_at": "2023-11-07T05:31:56Z",
  "completed_at": "2023-11-07T05:31:56Z",
  "created_at": "2023-11-07T05:31:56Z",
  "updated_at": "2023-11-07T05:31:56Z",
  "metrics": {},
  "stats": {
    "total_cost": 123,
    "total_errors": 123,
    "total_done": 123
  },
  "created_by_id": "<string>"
}

{
  "message": "<string>",
  "code": 123,
  "details": [
    "<unknown>"
  ]
}

Runs

Execute a run

Executes cells at the requested scope. FULL, PARTIAL, ROW and COLUMN queue the matching cells (a task column also queues its linked evaluator and metric cells) and start the orchestrator, which also re-runs failed cells of every row it picks up. A run that is already queued or running rejects new execution until it finishes or is cancelled. CELL re-runs one task or evaluator cell in place. Re-executing a terminal run does not require creating a new run.

POST

experiments

{experiment_id}

runs

{run_id}

execute

Core - Execute a run

curl 'https://my.orq.ai/v3/experiments/<experiment_id>/runs/<run_id>/execute' \
  --header 'Authorization: Bearer $ORQ_API_KEY' \
  --header 'Content-Type: application/json' \
  --data '{
    "scope": "EXECUTE_SCOPE_FULL",
    "constraints": { "llm": { "concurrency": 5, "timeout": "300s" } }
  }'

import { Orq } from '@orq-ai/node';

const client = new Orq({ apiKey: process.env.ORQ_API_KEY });

const run = await client.experiments.executeRun({
  experimentId: '<experiment_id>',
  runId: '<run_id>',
  scope: 'EXECUTE_SCOPE_FULL',
  constraints: { llm: { concurrency: 5, timeout: '300s' } },
});

console.log(run.status);

import os
from orq_ai_sdk import Orq

client = Orq(api_key=os.environ["ORQ_API_KEY"])

run = client.experiments.execute_run(
    experiment_id="<experiment_id>",
    run_id="<run_id>",
    scope="EXECUTE_SCOPE_FULL",
    constraints={"llm": {"concurrency": 5, "timeout": "300s"}},
)

print(run.status)

{
  "run_id": "<string>",
  "experiment_id": "<string>",
  "key": "<string>",
  "status": "RUN_STATUS_UNSPECIFIED",
  "dataset_id": "<string>",
  "row_count": 123,
  "cell_status_counts": {
    "not_run": 123,
    "queued": 123,
    "running": 123,
    "completed": 123,
    "failed": 123,
    "canceled": 123
  },
  "columns": [
    {
      "column_id": "<string>",
      "key": "<string>",
      "display_name": "<string>",
      "position": 123,
      "active": true,
      "prompt": {
        "prompt_config": {},
        "messages_source": "MESSAGES_SOURCE_UNSPECIFIED",
        "engine": "<string>",
        "tools": [
          {}
        ],
        "knowledge_bases": [
          {}
        ],
        "metadata": {}
      },
      "agent": {
        "agent_key": "<string>",
        "messages_source": "MESSAGES_SOURCE_UNSPECIFIED",
        "messages": [
          {}
        ],
        "engine": "<string>",
        "parent_id": "<string>",
        "agent_tag": "<string>"
      },
      "openresponse": {},
      "evaluator": {
        "evaluator_id": "<string>",
        "evaluator_name": "<string>",
        "evaluator_type": "<string>",
        "evaluator_output_type": "<string>"
      },
      "review": {
        "human_eval_id": "<string>",
        "human_eval_name": "<string>",
        "human_eval_type": "<string>"
      },
      "dataset": {
        "column_type": "<string>"
      },
      "metric": {
        "metric_key": "<string>"
      },
      "job_output": {
        "job_name": "<string>"
      },
      "width": 123
    }
  ],
  "settings": {
    "concurrency": 123,
    "timeout": "<string>",
    "max_turns": 123,
    "hidden_column_ids": [
      "<string>"
    ]
  },
  "started_at": "2023-11-07T05:31:56Z",
  "completed_at": "2023-11-07T05:31:56Z",
  "created_at": "2023-11-07T05:31:56Z",
  "updated_at": "2023-11-07T05:31:56Z",
  "metrics": {},
  "stats": {
    "total_cost": 123,
    "total_errors": 123,
    "total_done": 123
  },
  "created_by_id": "<string>"
}

{
  "message": "<string>",
  "code": 123,
  "details": [
    "<unknown>"
  ]
}

Related guide: Experiments guide. See the Experiments guide for a walkthrough with examples.

Authorizations

Authorization

string

header

required

Bearer authentication header of the form Bearer <token>, where <token> is your auth token.

Path Parameters

experiment_id

string

required

ID of the experiment.

run_id

string

required

ID of the run.

Body

application/json

scope

enum<string>

required

What to execute. EXECUTE_SCOPE_FULL: every executable cell. EXECUTE_SCOPE_PARTIAL: like full, but skips completed and running cells. EXECUTE_SCOPE_ROW: the cells of the rows in targets. EXECUTE_SCOPE_COLUMN: the cells of the columns in targets. EXECUTE_SCOPE_CELL: each combination of a target row and a target column. Targets also narrow full and partial executions.

Available options:

EXECUTE_SCOPE_UNSPECIFIED,

EXECUTE_SCOPE_CELL,

EXECUTE_SCOPE_ROW,

EXECUTE_SCOPE_COLUMN,

EXECUTE_SCOPE_PARTIAL,

EXECUTE_SCOPE_FULL

targets

object[]

Rows and columns to execute. Required for row, column and cell scope.

Show child attributes

constraints

object

Limits for this execution, saved into the run's settings.

Show child attributes

Response

A run: one pass of the experiment's columns over a copy of its dataset rows.

run_id

string

Run ID (ULID).

experiment_id

string

Experiment the run belongs to.

key

string

Run name. Run 1 for the first run; Run plus the last four characters of the run ID when a run is created without a key.

status

enum<string>

Run state. RUN_STATUS_DRAFT: created, never executed. RUN_STATUS_RUNNING: executing. RUN_STATUS_COMPLETED: no rows left to execute, although individual cells may have failed. RUN_STATUS_FAILED: execution stopped on an internal error. RUN_STATUS_CANCELED: stopped by Cancel a run. RUN_STATUS_QUEUED counts as active, like running.

Available options:

RUN_STATUS_UNSPECIFIED,

RUN_STATUS_DRAFT,

RUN_STATUS_QUEUED,

RUN_STATUS_RUNNING,

RUN_STATUS_COMPLETED,

RUN_STATUS_FAILED,

RUN_STATUS_CANCELED

dataset_id

string

Dataset the rows were copied from, taken from the experiment when the run was created. Empty for evaluatorq uploads.

row_count

integer<int32>

Number of rows in the run.

cell_status_counts

object

Number of cells in each status.

Show child attributes

columns

object[]

Columns of the run, hidden ones included with active false. Get, create and update order them as dataset, task, job output, metric, evaluator and review columns.

Show child attributes

settings

object | null

Null on runs created without settings, such as evaluatorq uploads.

Show child attributes

started_at

string<date-time> | null

Time the latest execution started. Null until the run is executed.

completed_at

string<date-time> | null

Time the run completed or was canceled. Null until then, and kept when the run is executed again.

created_at

string<date-time>

Creation time.

updated_at

string<date-time>

Time of the last change to the run.

metrics

object

Average evaluator results per task column, keyed {task_column_id}_{evaluator_id} (the run ID replaces the task column ID for run-level values). Only returned by List runs with include_metrics; categorical evaluators are left out.

Show child attributes

stats

object

Execution totals of the run.

Show child attributes

created_by_id

string

ID of the user or API key that created the run.

Orq SDK

Terminal & Agents

Terraform

Platform APIs

Management APIs

Execute a run

Authorizations

Path Parameters

Body

Response