HomepageAdmin PortalSign up
API Reference

Client libraries

We recommend to use our official client libraries to interact with the orq.ai API

Installation

pip install orq-ai-sdk

# npm
npm install @orq-ai/node

# yarn
yarn add @orq-ai/node

Usage

You can get your workspace API keys from the settings section in your orq.ai workspace. https://my.orq.ai/<workspace>/settings/developers

Initialize the orq.ai client with your API key:

import os

from orq_ai_sdk import OrqAI

client = OrqAI(
	 api_key=os.environ.get("ORQ_API_KEY", "__API_KEY__"),
 	 environment="production"
)

import os

from orq_ai_sdk import AsyncOrqAI

client = AsyncOrqAI(
	api_key=os.environ.get("ORQUESTA_API_KEY", "__API_KEY__"),
  environment="production"
)

import { createClient } from '@orq-ai/node';

const client = createClient({
  apiKey: 'orq-api-key',
  environment: 'production',
});

Set the contact after the client has been initialized.

Setting a contact identifier in the client is crucial for precisely tracking individual user requests. This enhances personalized user experiences and improves security by ensuring accurate user activity monitoring and efficient management.

client.set_contact(id=2024)

client.setContact({id: 2024})

Deployments

The Deployments API delivers text outputs, images, or tool calls based on the configuration established within orq.ai for your deployments. Additionally, this API supports streaming. It is highly recommended that you use the code snippets from the orq.ai Admin panel to ensure ease of use and minimize errors.

Invoke a deployment

invoke()

generation = client.deployments.invoke(
    key="customer_service",
    context={"environments": "production", "country": "NLD"},
    inputs={"firstname": "John", "city": "New York"},
    metadata={"customer_id": "Qwtqwty90281"},
)

print(generation.choices[0].message.content)

generation = await client.deployments.invoke(
    key="customer_service",
    context={"environments": "production", "country": "NLD"},
    inputs={"firstname": "John", "city": "New York"},
    metadata={"customer_id": "Qwtqwty90281"},
)

print(generation.choices[0].message.content)

const deployment = await client.deployments.invoke({
  key: 'customer_service',
  context: { environments: 'production', country: 'NLD' },
  inputs: { firstname: 'John', city: 'New York' },
  metadata: { customer_id: 'Qwtqwty90281' },
});

console.log(deployment?.choices[0].message.content);

invoke_with_stream()

stream = client.deployments.invoke_with_stream(
    key="customer_service",
    context={"environments": "production", "country": "NLD"},
    inputs={"firstname": "John", "city": "New York"},
    metadata={"customer_id": "Qwtqwty90281"},
)

for chunk in stream:
    if chunk.is_final:
        print("Stream is finished")

    print(chunk.choices[0].message.content)


stream = client.deployments.invoke_with_stream(
    key="customer_service",
    context={ "environments": "production", "country": "NLD" },
    inputs={ "firstname": "John", "city": "New York" },
    metadata={ "customer_id": "Qwtqwty90281" },
)

async for chunk in stream:
    if chunk.is_final:
        print("Stream is finished")

		print(chunk.choices[0].message.content)

const stream = client.deployments.invokeWithStream({
  key: 'customer_service',
  context: { environments: 'production', country: 'NLD' },
  inputs: { firstname: 'John', city: 'New York' },
  metadata: { customer_id: 'Qwtqwty90281' },
});

for await (const chunk of stream) {
  console.log(chunk.choices[0].message.content);
}

Adding messages as part of your request

If you are using the invoke method, you can include messages in your request to the model. The messages property allows you to combine chat history with the prompt configuration in orq.ai, or to directly send messages to the model if you manage the prompt in your code.

generation = client.deployments.invoke(
    key="customer_service",
    context={
        "language": [],
        "environments": []
    },
    metadata={
        "custom-field-name": "custom-metadata-value"
    },
    inputs={"firstname": "John", "city": "New York"},
    messages=[{
        "role": "user",
        "content": "A customer is asking about the latest software update features. Generate a detailed and informative response highlighting the key new features and improvements in the latest update.",
    }]
)

print(generation.choices[0].message.content)

generation = await client.deployments.invoke(
    key="customer_service",
    context={
        "language": [],
        "environments": []
    },
    metadata={
        "custom-field-name": "custom-metadata-value"
    },
    inputs={"firstname": "John", "city": "New York"},
    messages=[{
        "role": "user",
        "content": "A customer is asking about the latest software update features. Generate a detailed and informative response highlighting the key new features and improvements in the latest update.",
    }]
)

print(generation.choices[0].message.content)

const generation = await client.deployments.invoke({
  key: 'customer_service',
  messages: [
    {
      role: 'user',
      content:
        'A customer is asking about the latest software update features. Generate a detailed and informative response highlighting the key new features and improvements in the latest update.',
    },
  ],
  context: { environments: 'production', country: 'NLD' },
  inputs: { firstname: 'John', city: 'New York' },
  metadata: { customer_id: 'Qwtqwty90281' },
});

console.log(generation?.choices[0].message.content);

Adding prefix messages

Prefix messages are ways to invoke your deployment with extra historical and contextual messages. Those messages will be sent before the ones configured user and assistantmessages in your Deployment. You can define a role to use for the prefix messages.

Using Prefix Messages is useful to send reminders to the model about the current conversation at hand or preferences to personalize the response.

generation = client.deployments.invoke(
    key="customer_service",
    context={
        "language": [],
        "environments": []
    },
    metadata={
        "custom-field-name": "custom-metadata-value"
    },
    inputs={"firstname": "John", "city": "New York"},
    prefix_messages=[{
        "role": "user",
        "content": "This user is already familiar with our software, reply as if they are an advanced operator.",
    }]
)

print(generation.choices[0].message.content)

generation = await client.deployments.invoke(
    key="customer_service",
    context={
        "language": [],
        "environments": []
    },
    metadata={
        "custom-field-name": "custom-metadata-value"
    },
    inputs={"firstname": "John", "city": "New York"},
    prefix_messages=[{
        "role": "user",
        "content": "This user is already familiar with our software, reply as if they are an advanced operator.",
    }]
)

print(generation.choices[0].message.content)

const generation = await client.deployments.invoke({
  key: 'customer_service',
  prefix_messages: [
    {
      role: 'user',
      content:
        'This user is already familiar with our software, reply as if they are an advanced operator.',
    },
  ],
  context: { environments: 'production', country: 'NLD' },
  inputs: { firstname: 'John', city: 'New York' },
  metadata: { customer_id: 'Qwtqwty90281' },
});

console.log(generation?.choices[0].message.content);

Logging metrics to the deployment

After invoking or streaming a deployment, you can use the add_metrics method to add information to the deployment.

generation.add_metrics(
    chain_id="c4a75b53-62fa-401b-8e97-493f3d299316",
    conversation_id="ee7b0c8c-eeb2-43cf-83e9-a4a49f8f13ea",
    user_id="e3a202a6-461b-447c-abe2-018ba4d04cd0",
    feedback={"score": 100},
    metadata={
        "custom": "custom_metadata",
        "chain_id": "ad1231xsdaABw",
    }
)

await generation.add_metrics(
    chain_id="c4a75b53-62fa-401b-8e97-493f3d299316",
    conversation_id="ee7b0c8c-eeb2-43cf-83e9-a4a49f8f13ea",
    user_id="e3a202a6-461b-447c-abe2-018ba4d04cd0",
    feedback={"score": 100},
    metadata={
        "custom": "custom_metadata",
        "chain_id": "ad1231xsdaABw",
    }
)

generation.addMetrics({
  chain_id: 'c4a75b53-62fa-401b-8e97-493f3d299316',
  conversation_id: 'ee7b0c8c-eeb2-43cf-83e9-a4a49f8f13ea',
  user_id: 'e3a202a6-461b-447c-abe2-018ba4d04cd0',
  feedback: { score: 100 },
  metadata: {
    custom: 'custom_metadata',
    chain_id: 'ad1231xsdaABw',
  },
});

Invoke the model with extra_params

Currently, orq.ai supports around 100 models. As innovation in the AI industry moves fast and LLM providers continually improve their APIs, new parameters are available with each new release.

To keep up, we have decided to introduce extra_params, with which you can send to the provider the model parameters that are not supported in orq.ai.

Here for instance we add the field seed into extra_params. OpenAI implemented the seed parameter alongside their Completion API to enable determinism in generations. Adding this field in the extra_params will forward the parameter to the OpenAI API, this way you can always be up to date with the latest implementation and feature-set.

generation = client.deployments.invoke(
  key="customer_service",
  extra_params={
  	"seed": "1677652288",
    "logit_bias": 50
  }
);

print(generation.choices[0].message.content)

generation = await client.deployments.invoke(
  key="customer_service",
  extra_params={
  	"seed": "1677652288",
    "logit_bias": 50
  }
);

print(generation.choices[0].message.content)

const generation = client.deployments.invoke({
	key: "customer_service",
	extra_params: {
		seed: "1677652288",
		logit_bias: 50,
	},
});

console.log(generation.choices[0].message.content);

🚧

Each provider has its own set of parameters, and we recommend reading their documentation carefully. Keep in mind that any extra parameter (extra_params) included in the request to our API will overwrite the parameters configured in the deployment.

Get deployment configuration

get_config()

prompt_config = client.deployments.get_config(
    key="customer_service",
    context={ "environments": "production", "country": "NLD" },
    inputs={ "firstname": "John", "city": "New York" },
    metadata={ "customer_id": "Qwtqwty90281" },
)

print(prompt_config.to_dict())

prompt_config = await client.deployments.get_config(
    key="customer_service",
    context={ "environments": "production", "country": "NLD" },
    inputs={ "firstname": "John", "city": "New York" },
    metadata={ "customer_id": "Qwtqwty90281" },
)

print(prompt_config.to_dict())

const promptConfig = await client.deployments.getConfig({
  key: 'customer_service',
  context: { environments: 'production', country: 'NLD' },
  inputs: { firstname: 'John', city: 'New York' },
  metadata: { customer_id: 'Qwtqwty90281' },
});

console.log(promptConfig);

Logging metrics to the deployment configuration

After getting the configuration of a deployment, you can use the add_metrics method to add information to the deployment.

generation.add_metrics(
    chain_id="c4a75b53-62fa-401b-8e97-493f3d299316",
    conversation_id="ee7b0c8c-eeb2-43cf-83e9-a4a49f8f13ea",
    user_id="e3a202a6-461b-447c-abe2-018ba4d04cd0",
    feedback={"score": 100},
    metadata={
        "custom": "custom_metadata",
        "chain_id": "ad1231xsdaABw",
    },
    usage={
        "prompt_tokens": 100,
        "completion_tokens": 900,
        "total_tokens": 1000,
    },
    performance={
        "latency": 9000,
        "time_to_first_token": 250,
    },
    messages=[
        {
            "role": "user",
            "message": "A customer is asking about the latest software update features. Generate a detailed and informative response highlighting the key new features and improvements in the latest update.",
        },
    ],
)

await generation.add_metrics(
    chain_id="c4a75b53-62fa-401b-8e97-493f3d299316",
    conversation_id="ee7b0c8c-eeb2-43cf-83e9-a4a49f8f13ea",
    user_id="e3a202a6-461b-447c-abe2-018ba4d04cd0",
    feedback={"score": 100},
    metadata={
        "custom": "custom_metadata",
        "chain_id": "ad1231xsdaABw",
    },
    usage={
        "prompt_tokens": 100,
        "completion_tokens": 900,
        "total_tokens": 1000,
    },
    performance={
        "latency": 9000,
        "time_to_first_token": 250,
    },
    messages=[
        {
            "role": "user",
            "message": "A customer is asking about the latest software update features. Generate a detailed and informative response highlighting the key new features and improvements in the latest update.",
        },
    ],
)

generation.addMetrics({
  chain_id: 'c4a75b53-62fa-401b-8e97-493f3d299316',
  conversation_id: 'ee7b0c8c-eeb2-43cf-83e9-a4a49f8f13ea',
  user_id: 'e3a202a6-461b-447c-abe2-018ba4d04cd0',
  feedback: { score: 100 },
  metadata: {
    custom: 'custom_metadata',
    chain_id: 'ad1231xsdaABw',
  },
  usage: {
    prompt_tokens: 100,
    completion_tokens: 900,
    total_tokens: 1000,
  },
  performance: {
    latency: 9000,
    time_to_first_token: 250,
  },
  messages: [
    {
      role: 'user',
      message:
        'A customer is asking about the latest software update features. Generate a detailed and informative response highlighting the key new features and improvements in the latest update.',
    },
  ],
});

Logging LLM responses

Whether you use the get_config or invoke, you can log the model generations to the deployment. Here are some examples of how to perform this action.

Logging the completion choices the model generated for the input prompt

generation.add_metrics(
    choices=[
        {
            "index": 0,
            "finish_reason": "assistant",
            "message": {
                "role": "assistant",
                "content": "Dear customer: Thank you for your interest in our latest software update! We're excited to share with you the new features and improvements we've rolled out. Here's what you can look forward to in this update",
            },
        },
    ]
)

await generation.add_metrics(
    choices=[
        {
            "index": 0,
            "finish_reason": "assistant",
            "message": {
                "role": "assistant",
                "content": "Dear customer: Thank you for your interest in our latest software update! We're excited to share with you the new features and improvements we've rolled out. Here's what you can look forward to in this update",
            },
        },
    ]
)

generation.addMetrics({
  choices: [
    {
      index: 0,
      finish_reason: 'stop',
      message: {
        role: 'assistant',
        content:
          "Dear customer: Thank you for your interest in our latest software update! We're excited to share with you the new features and improvements we've rolled out. Here's what you can look forward to in this update",
      },
    },
  ],
});

Logging the completion choices the model generated for the input prompt

You can save the images generated by the model in orq.ai. If the image format is base64 we always store it as a png.

generation.add_metrics(
    choices=[
        {
            "index": 0,
            "finish_reason": 'stop',
            "message": {
                "role": "assistant",
                "url": "<image_url>"
            },
        },
    ],
)

await generation.add_metrics(
    choices=[
        {
            "index": 0,
            "finish_reason": 'stop',
            "message": {
                "role": "assistant",
                "url": "<image_url>"
            },
        },
    ],
)

generation.addMetrics({
  choices: [
    {
      index: 0,
      finish_reason: 'stop',
      message: {
        role: 'assistant',
        url: 'https://oaidalleapiprodscus.blob.core.windows.net/private/org-HunR6LApWxZ7z1JS4w7Ot2ux/user-wB8Cy1SbfbQQsj6tw7hljqgU/img-nEQPFbZ9fvkPMSM5YkCEXCdv.png?st=2024-02-13T19%3A09%3A12Z&se=2024-02-13T21%3A09%3A12Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2024-02-13T17%3A31%3A27Z&ske=2024-02-14T17%3A31%3A27Z&sks=b&skv=2021-08-06&sig=3B3mlUIlVj8A1nKfyD2e1YEaR/RsO1dSpCCesI/tC0s%3D',
      },
    },
  ],
});

Logging the output of the tool calls

generation.add_metrics(
  choices=[
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": None,
        "tool_calls": [
          {
            "type": "function",
            "id": "call_pDBPMMacPXOtoWhTWibW1D94",
            "function": {
              "name": "get_weather",
              "arguments": '{"location":"San Francisco, CA"}',
            },
          },
        ],
      },
      "finish_reason": 'tool_calls',
    }
  ]
)

await generation.add_metrics(
  choices=[
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": None,
        "tool_calls": [
          {
            "type": "function",
            "id": "call_pDBPMMacPXOtoWhTWibW1D94",
            "function": {
              "name": "get_weather",
              "arguments": '{"location":"San Francisco, CA"}',
            },
          },
        ],
      },
      "finish_reason": 'tool_calls',
    }
  ]
)

generation.addMetrics({
  choices: [
    {
      index: 0,
      message: {
        role: 'assistant',
        content: null,
        tool_calls: [
          {
            type: 'function',
            id: 'call_pDBPMMacPXOtoWhTWibW1D94',
            function: {
              name: 'get_weather',
              arguments: '{"location":"San Francisco, CA"}',
            },
          },
        ],
      },
      finish_reason: 'tool_calls',
    },
  ],
});