Creating an Experiment

Prerequisites

Before creating an Experiment, you need a Dataset. This dataset contains the Inputs, Messages, and Expected Outputs used for running an Experiment.

• Inputs – Variables that can be used in the prompt message, e.g., {{firstname}}.
• Messages – The prompt template, structured with system, user, and assistant roles.
• Expected Outputs – Reference responses that evaluators use to compare against newly generated outputs.

To get experiments ready, make sure you have models available by adding them to your Model Garden.

👍

You don’t need to include all three entities when uploading a dataset. Depending on your experiment, you can choose to include only inputs, messages, or expected outputs as needed. For example, you can create a dataset with just inputs.

Creating an Experiment

To create an Experiment, head to your Orq.ai Panel:

• Choose a Project and Folder and select the + button.
• Choose Experiment

Afterwards, select a Dataset to use as a base for your Experiment and choose one or multiple models.

You'll be then taken to the Report view where you can configure Dataset entries and models before running the experiment.

Configuring Experiment

Data Entry Configuration

The left side of the table shows the loaded Dataset entries. Each entry is shown as a row and will be executed separately with each configured prompt.

You can add new entries to be tested during experiment by using the Add Row button.

Each entry's Inputs, Messages and Expected Outputs can be edited independently by selecting a cell.

Prompt Configuration

Your chosen prompts are displayed as separate column within the Response section.

Prompts are assigned a corresponding letter (see Aand B above) to identify their performance and Evaluators results.

To add a new prompt, head to the right of the table, select Add a Column and choose Prompt.

Select the Prompt Name to open the Prompt panel and Configure the Prompt Template.

👍

There are 2 ways to configure your prompt: Using the 'messages' field in the dataset or by using the Response columns

📘

To learn more about Prompt Template Configuration, see Configuring a Prompt in Experiments

Configuring Evaluators

You can choose to add Evaluators to automatically validate and compare outputs.

To add a new Evaluator, head to the right of the table, select Add a Column and choose Evaluator.

📘

To learn more about Evaluators in Experiments, see Evaluators in Experiments