Skip to main content

​
What are curated datasets

A normal dataset consists of a prompt and an attached reference. Within Orq.ai you have the ability to create curated datasets. Curated datasets are human-evaluated input and output sets. In other words, you have a set with a prompt with an expected output. These curated datasets can be used to fine-tune a model. Because you’re providing a model with specific examples, you can help tune its behavior more precisely to your needs.

​
Creating curated datasets

Within any module, by choosing the Logs tab and then selecting a single Log, the Feedback panel will be displayed on the right. A domain expert can review the logs to verify and correct the output. To add a correction, find the Add correction button:

The Add correction button is below Assistant response.

Choosing to add a correction will open a new Correction message in which you can edit manually the response provided by the model. Once you have finished editing, select Save to use that correction.

The corrected text and correction will appear next to one another, the correction is displayed in green.

Once chosen and eventually corrected, you can choose to add the response to a Curated Dataset. To do so select the Add to Dataset icon at the top-right of the response: Here you will see a list of datasets you can add the log to, to create a new Dataset, see Creating a Dataset.

You can choose to replace the inputs used during generation (recommended).

​
Using a curated dataset with Experiments

Another common use case is using curated datasets within the Experiment module. When Creating an Experiment, import the curated dataset, attach an Evaluator and see which model and/or prompt scores the best in relation to the Curated Dataset. To learn more about using datasets in experiments, see Creating an Experiment.