Orq.ai Documentation - AI Gateway & LLM Collaboration Platform

Use Cases

Connecting models to live data (databases, calendars, internal APIs) without prompt hacks.
Agents that take actions on behalf of users: create tickets, send emails, run queries.
Multi-step workflows where the model decides which tools to invoke and in what order.
Replacing brittle regex parsing with structured function calls for data extraction.

Quick Start

Enable AI models to call external functions with structured parameters.

curl -X POST https://api.orq.ai/v3/router/responses \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o",
    "input": "What is the weather in NYC?",
    "tools": [{
      "type": "function",
      "name": "get_weather",
      "description": "Get current weather for a location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": { "type": "string", "description": "City and state" },
          "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] }
        },
        "required": ["location"]
      }
    }],
    "tool_choice": "auto"
  }'

Configuration

Tool Definition

Responses API (/v3/router/responses): flat shape:

Parameter	Type	Required	Description
`type`	`"function"`	Yes	Tool type
`name`	string	Yes	Function identifier
`description`	string	Yes	What the function does
`parameters`	object	Yes	JSON Schema for parameters

Chat Completions (/v3/router/chat/completions): nested function wrapper:

Parameter	Type	Required	Description
`type`	`"function"`	Yes	Tool type
`function.name`	string	Yes	Function identifier
`function.description`	string	Yes	What the function does
`function.parameters`	object	Yes	JSON Schema for parameters

Tool Choice Options

Value	Behavior
`"auto"`	Model decides when to use tools
`"none"`	Disable tool usage
`"required"`	Force tool usage
`{type: "function", function: {name: "tool_name"}}`	Force specific tool

Tool Message Format

This format applies to the Chat Completions endpoint (/v3/router/chat/completions). On the Responses API, tool results use type: "function_call_output", call_id, and output instead.

When providing tool results back to the model, use the tool role:

Parameter	Type	Required	Description
`role`	`"tool"`	Yes	Message role for tool results
`tool_call_id`	string \| null	Yes	ID of the tool call being responded to
`content`	string	Yes	JSON-stringified result of the tool execution

The tool_call_id can be null in certain scenarios, such as when tool results are being provided without a corresponding tool call from the model, or when working with providers that don’t require tool call IDs.

Code examples

curl -X POST https://api.orq.ai/v3/router/responses \
  -H "Authorization: Bearer $ORQ_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o",
    "input": "What is the weather like in San Francisco?",
    "tools": [
      {
        "type": "function",
        "name": "get_weather",
        "description": "Get the current weather for a location",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "The city and state, e.g. San Francisco, CA"
            },
            "unit": {
              "type": "string",
              "enum": ["celsius", "fahrenheit"],
              "description": "The temperature unit"
            }
          },
          "required": ["location"]
        }
      }
    ],
    "tool_choice": "auto"
  }'

Function Execution Patterns

Basic Tool Handler

class ToolHandler {
  constructor() {
    this.tools = new Map();
  }

  register(name, func, schema) {
    this.tools.set(name, { func, schema });
  }

  async execute(toolCall) {
    const tool = this.tools.get(toolCall.function.name);
    if (!tool) throw new Error(`Unknown tool: ${toolCall.function.name}`);

    const args = JSON.parse(toolCall.function.arguments);
    const result = await tool.func(args);

    return {
      role: "tool",
      tool_call_id: toolCall.id,
      content: JSON.stringify(result),
    };
  }
}

// Stubs - replace with your actual implementations
const weatherSchema = { type: "object", properties: { location: { type: "string" } }, required: ["location"] };
const searchSchema = { type: "object", properties: { query: { type: "string" } }, required: ["query"] };
const getWeatherAPI = async ({ location }) => ({ temperature: 72, condition: "sunny", location });
const searchWebAPI = async ({ query }) => ({ results: [`Result for: ${query}`] });
const response = { choices: [{ message: { tool_calls: [] } }] };

const handler = new ToolHandler();
handler.register("get_weather", getWeatherAPI, weatherSchema);
handler.register("search_web", searchWebAPI, searchSchema);

// Execute tool calls
const toolResults = await Promise.all(
  response.choices[0].message.tool_calls.map((call) => handler.execute(call)),
);

Parallel Tool Execution

import asyncio
import json

async def get_weather_async(args): return {"temperature": 72, "condition": "sunny"}  # replace with your weather API
async def search_products_async(args): return {"products": []}  # replace with your product search
async def check_inventory_async(args): return {"in_stock": True}  # replace with your inventory API

# Replace these with your actual API call and conversation history
response = None  # e.g. client.chat.completions.create(model=..., messages=messages, tools=[...])
messages: list = []

async def execute_tools_parallel(tool_calls):
    async def execute_single_tool(tool_call):
        function_name = tool_call.function.name
        arguments = json.loads(tool_call.function.arguments)

        # Route to appropriate function
        if function_name == "get_weather":
            result = await get_weather_async(arguments)
        elif function_name == "search_products":
            result = await search_products_async(arguments)
        elif function_name == "check_inventory":
            result = await check_inventory_async(arguments)
        else:
            result = {"error": f"Unknown function: {function_name}"}

        return {
            "role": "tool",
            "tool_call_id": tool_call.id,
            "content": json.dumps(result)
        }

    # Execute all tools concurrently
    results = await asyncio.gather(
        *[execute_single_tool(call) for call in tool_calls]
    )

    return results

# Usage
if response.choices[0].message.tool_calls:
    tool_results = await execute_tools_parallel(
        response.choices[0].message.tool_calls
    )

    # Add to conversation
    messages.extend(tool_results)

Advanced Use Cases

Database Integration

sanitize_sql = lambda q: q  # replace with your SQL sanitizer

def get_db_connection():
    pass  # replace with your database client, e.g. from myapp.db import get_db_connection

db = get_db_connection()

tools = [
    {
        "type": "function",
        "function": {
            "name": "query_database",
            "description": "Query the customer database",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {
                        "type": "string",
                        "description": "SQL query to execute"
                    },
                    "limit": {
                        "type": "integer",
                        "description": "Maximum number of results"
                    }
                },
                "required": ["query"]
            }
        }
    }
]

async def query_database(args):
    query = sanitize_sql(args["query"])
    limit = args.get("limit", 10)

    results = await db.execute(f"{query} LIMIT {limit}")
    return {"results": results.fetchall()}

API Integration

const apiTools = [
  {
    type: "function",
    function: {
      name: "send_email",
      description: "Send an email to a recipient",
      parameters: {
        type: "object",
        properties: {
          to: { type: "string", description: "Email address" },
          subject: { type: "string", description: "Email subject" },
          body: { type: "string", description: "Email content" },
        },
        required: ["to", "subject", "body"],
      },
    },
  },
  {
    type: "function",
    function: {
      name: "create_calendar_event",
      description: "Create a calendar event",
      parameters: {
        type: "object",
        properties: {
          title: { type: "string" },
          start_time: { type: "string", format: "date-time" },
          duration: { type: "integer", description: "Duration in minutes" },
          attendees: { type: "array", items: { type: "string" } },
        },
        required: ["title", "start_time"],
      },
    },
  },
];

const emailAPI = { send: async (args: Record<string, unknown>) => ({ messageId: "msg_001" }) }; // replace with your email client
const calendarAPI = { createEvent: async (args: Record<string, unknown>) => ({ eventId: "evt_001" }) }; // replace with your calendar client

const executeApiTool = async (toolCall) => {
  const { name } = toolCall.function;
  const args = JSON.parse(toolCall.function.arguments);

  switch (name) {
    case "send_email":
      return await emailAPI.send(args);
    case "create_calendar_event":
      return await calendarAPI.createEvent(args);
    default:
      throw new Error(`Unknown API tool: ${name}`);
  }
};

Multi-Steps Workflows

from openai import AsyncOpenAI
import os
import json

client = AsyncOpenAI(
    api_key=os.environ.get("ORQ_API_KEY"),
    base_url="https://api.orq.ai/v3/router",
)

class WorkflowEngine:
    def __init__(self):
        self.tools = {}
        self.conversation = []

    def register_tool(self, name, func, schema):
        self.tools[name] = {"func": func, "schema": schema}

    async def execute_workflow(self, initial_prompt, max_steps=10):
        self.conversation = [{"role": "user", "content": initial_prompt}]

        for step in range(max_steps):
            response = await client.chat.completions.create(
                model="openai/gpt-4o",
                messages=self.conversation,
                tools=[v["schema"] for v in self.tools.values()],
                tool_choice="auto"
            )

            self.conversation.append(response.choices[0].message)

            # Check if tools need to be executed
            if response.choices[0].message.tool_calls:
                for tool_call in response.choices[0].message.tool_calls:
                    result = await self.execute_tool(tool_call)
                    self.conversation.append(result)
            else:
                # No tools called, workflow complete
                return response.choices[0].message.content

        return "Workflow exceeded maximum steps"

    async def execute_tool(self, tool_call):
        tool_name = tool_call.function.name
        arguments = json.loads(tool_call.function.arguments)

        if tool_name in self.tools:
            result = await self.tools[tool_name]["func"](arguments)
        else:
            result = {"error": f"Unknown tool: {tool_name}"}

        return {
            "role": "tool",
            "tool_call_id": tool_call.id,
            "content": json.dumps(result)
        }

Error Handling

const validateArgs = (name: string, args: Record<string, unknown>) => ({ valid: true, errors: [] as string[] }); // replace with your validator
const executeFunction = async (name: string, args: Record<string, unknown>): Promise<unknown> => ({}); // replace with your dispatcher

const safeToolExecution = async (toolCall) => {
  try {
    const args = JSON.parse(toolCall.function.arguments);

    // Validate arguments
    const validation = validateArgs(toolCall.function.name, args);
    if (!validation.valid) {
      return {
        role: "tool",
        tool_call_id: toolCall.id,
        content: JSON.stringify({
          error: "Invalid arguments",
          details: validation.errors,
        }),
      };
    }

    // Execute with timeout
    const result = await Promise.race([
      executeFunction(toolCall.function.name, args),
      new Promise((_, reject) =>
        setTimeout(() => reject(new Error("Tool execution timeout")), 30000),
      ),
    ]);

    return {
      role: "tool",
      tool_call_id: toolCall.id,
      content: JSON.stringify(result),
    };
  } catch (error) {
    console.error(`Tool execution failed: ${error.message}`);

    return {
      role: "tool",
      tool_call_id: toolCall.id,
      content: JSON.stringify({
        error: "Tool execution failed",
        message: error.message,
      }),
    };
  }
};

Best Practices

Tool design

Use clear, descriptive function names.
Provide detailed parameter descriptions.
Include examples in descriptions.
Make tools idempotent when possible.

Schema design

{
  "type": "object",
  "properties": {
    "location": {
      "type": "string",
      "description": "City and state (e.g., 'San Francisco, CA')",
      "examples": ["New York, NY", "London, UK"]
    },
    "units": {
      "type": "string",
      "enum": ["metric", "imperial"],
      "description": "Temperature unit system",
      "default": "metric"
    }
  },
  "required": ["location"]
}

Security considerations

Never expose destructive operations directly.
Validate all inputs thoroughly.
Use allowlists for sensitive operations.
Implement proper authentication.
Log all tool executions.

Troubleshooting

Tool not being called

Check tool descriptions are clear.
Verify parameter schemas are correct.
Ensure tool_choice is set appropriately.
Try more explicit prompts. Invalid arguments
Validate JSON Schema thoroughly.
Add parameter examples.
Check required fields are marked.
Simplify complex parameter structures.

Execution failures

Implement proper error handling.
Add timeout protection.
Validate inputs before execution.
Return structured error messages.

Limitations

Limitation	Impact	Workaround
Tool limit	Max ~20 tools per request	Group related functions
Parameter size	Large schemas may fail	Simplify parameter structure
Execution time	Tools block response	Use async patterns
Error propagation	Failures can break workflow	Implement error recovery
Model differences	Varying tool calling quality	Test across models

Performance Optimization

Tool caching

class CachedToolExecutor {
  constructor() {
    this.cache = new Map();
    this.cacheTTL = 300000; // 5 minutes
  }

  getCacheKey(toolCall) {
    return `${toolCall.function.name}:${toolCall.function.arguments}`;
  }

  async executeFunction(toolCall) {
    throw new Error("executeFunction must be implemented in a subclass");
  }

  async execute(toolCall) {
    const key = this.getCacheKey(toolCall);
    const cached = this.cache.get(key);

    if (cached && Date.now() - cached.timestamp < this.cacheTTL) {
      return cached.result;
    }

    const result = await this.executeFunction(toolCall);
    this.cache.set(key, { result, timestamp: Date.now() });

    return result;
  }
}

Batch operations

# Pattern fragment: add client setup and get_weather stub before using
# Instead of multiple individual calls
def get_weather_batch(locations):
    return {loc: get_weather(loc) for loc in locations}

# Tool that accepts multiple inputs
get_weather_batch_tool = {
    "name": "get_weather_batch",
    "parameters": {
        "type": "object",
        "properties": {
            "locations": {
                "type": "array",
                "items": {"type": "string"}
            }
        }
    }
}

​Quick Start

​Configuration

​Tool Definition

​Tool Choice Options

​Tool Message Format

​Code examples

​Function Execution Patterns

​Basic Tool Handler

​Parallel Tool Execution

​Advanced Use Cases

​Database Integration

​API Integration

​Multi-Steps Workflows

​Error Handling

​Best Practices

​Tool design

​Schema design

​Security considerations

​Troubleshooting

​Limitations

​Performance Optimization

​Tool caching

​Batch operations

Quick Start

Configuration

Tool Definition

Tool Choice Options

Tool Message Format

Code examples

Function Execution Patterns

Basic Tool Handler

Parallel Tool Execution

Advanced Use Cases

Database Integration

API Integration

Multi-Steps Workflows

Error Handling

Best Practices

Tool design

Schema design

Security considerations

Troubleshooting

Limitations

Performance Optimization

Tool caching

Batch operations