Playgrounds

Orq.ai LLM Playground for quick experimentation and interacting with multiple Large Language Models (LLMs) to test hypotheses and new use cases.

Orq.ai's Playground is an experimentation playground where you can interact with multiple Large Language Models (LLMs), perform comparisons based on the LLM response, keep track of metrics such as costs and latency, and provide human feedback.

Setting up a LLM Playground

Create a new Playground. The playground is auto-saved, so you can easily come back where you left off or hand it over to a team member to continue.

Playground Tools

The playground comprises experiment blocks; you can add up to 6 blocks by clicking the add model button.

You can delete a block within the playground by clicking the delete button, clear a conversation within a block, and copy the entire response. The History feature shows you the playground memory; once enabled, you can view any number of past messages.

Configuring Playground Block

For your model, you can select a model by clicking on the Model selection dropdown (the prompt type will be displayed beside it: Chat, Completion, image)

To get a better response from the model, the playground offers a unique feature that helps configure the model by clicking on the gear icon. Some of these parameters include the number of words, frequency penalty, Temperature, Top K and Top P.

To input your prompts, use the prompt field within the block; type in your prompt and press the send button. You can sync chats in the playground to test the same prompt across different model configurations easily.

You can also create variables using opening and closing curly brackets {{ }}, place the variable key inside it within the prompt. To do this, you will have to create the variable, for example, {{category}} and automatically, it gets added to the sidebar. Edit the variable to your satisfaction in the input field. This is useful for RAG use cases and working with large pieces of context in your prompt engineering.

For each response in the playground, you can access its cost and latency.

Playground History

History is a feature in the Playground that functions as the Playground memory. It selects the number of past messages to include in each playground completion. Setting this number to 5 will include pairs of 5 user queries and assistant responses. This only applies to models that support chat history and helps to give the model context for new user queries.

In the Playground, when you send a new system prompt, it triggers a conversation restart. This means that any ongoing conversation will be reset, and only sending user messages will allow you to continue the chat while retaining the chat history.

Internal feedback

Feedback is very important, as it helps to know how the model performs based on its output. The playground uses "thumbs up" and "thumbs down" to provide feedback for each response, making it easy for internal team members or domain experts to add feedback. The collected feedback can then be used for model fine-tuning as a data set.

Logs

All the runs are recorded in the Logs tab in the Playground. Logs provide a detailed record of events and interactions with the LLM. The components of the logs include: Conversation, Request, Feedback, and Debug.

Function calling

Function calling is the process of invoking or executing a specific function within the model. Functions are pre-defined blocks of code that perform specific tasks or operations. When a function is called, the model executes the instructions for the desired actions.

Check out how to perform function calling in the Playground: Function Calling in Playground