Databricks integration

Databricks Model Serving hosts both external models and agents authored against the MLflow ResponsesAgent interface. Register the AI Gateway as an external model to reach 300+ models from inside Databricks, and instrument agents with OpenTelemetry to capture their LLM calls in Orq.ai.

AI Gateway

Serve any Orq.ai model through a Databricks serving endpoint, with cost tracking and fallbacks.

Observability

Export traces from agents running on Model Serving, with prompts, tokens, and cost.

Prerequisites

An Orq.ai account and API key
A Databricks workspace with Model Serving enabled
MLflow 3.11 or later, plus databricks-sdk and databricks-agents installed locally

Store the API key

Both sections below read the key from a Databricks secret. Keep it there rather than in an endpoint configuration, where anyone with view access on the endpoint can read it. Skip create_scope if the scope already exists:

from databricks.sdk import WorkspaceClient

workspace = WorkspaceClient()
workspace.secrets.create_scope(scope="orq")
workspace.secrets.put_secret(
    scope="orq", key="api_key", string_value="<ORQ_API_KEY>"
)

AI Gateway

Databricks external models accept any OpenAI-compatible endpoint through the custom provider, so the AI Gateway can back a Databricks serving endpoint. Set external_model.name to the model to serve, and point custom_provider_url at the AI Gateway:

import mlflow.deployments

client = mlflow.deployments.get_deploy_client("databricks")

client.create_endpoint(
    config={
        "name": "orq-gateway",
        "config": {
            "served_entities": [
                {
                    "name": "orq-gateway",
                    "external_model": {
                        "name": "openai/gpt-5.6-sol",
                        "provider": "custom",
                        "task": "llm/v1/chat",
                        "custom_provider_config": {
                            "custom_provider_url": "https://api.orq.ai/v3/router/chat/completions",
                            "bearer_token_auth": {
                                "token": "{{secrets/orq/api_key}}"
                            },
                        },
                    },
                }
            ]
        },
    },
)

Query it like any other Databricks serving endpoint. Calls appear in Traces with model, token usage, and cost:

response = client.predict(
    endpoint="orq-gateway",
    inputs={"messages": [{"role": "user", "content": "Hello"}]},
)
print(response["choices"][0]["message"]["content"])

Any enabled model works. Set external_model.name to its Orq.ai slug and see supported models. One endpoint serves one model, so create an endpoint per model.

Observability

Adding OpenTelemetry to an agent already running on Model Serving sends its LLM calls to Orq.ai, with full request and response content, token usage, and cost. There are four changes: add an exporter at module level, flush at the end of predict, declare three extra packages, and pass the endpoint and key at deploy time.

This covers agents deployed from code with agents.deploy(). Agents created in the no-code Agent Bricks builder run on an endpoint that Databricks manages, which exposes neither environment variables nor model dependencies, so they cannot be instrumented this way.

Add the exporter to the agent

Add this at module level in the agent file, above the agent class. It opens the connection to Orq.ai and attaches the instrumentor that records LLM calls. OpenAIInstrumentor records calls made through the OpenAI SDK, which is what WorkspaceClient().serving_endpoints.get_open_ai_client() returns. Agents built on another client library need the matching OpenInference instrumentor in its place, otherwise the agent deploys cleanly and produces no spans:

import os

from openinference.instrumentation.openai import OpenAIInstrumentor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

tracer_provider = TracerProvider()
tracer_provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint=os.environ["ORQ_OTEL_ENDPOINT"],
            headers={"Authorization": f"Bearer {os.environ['ORQ_API_KEY']}"},
        )
    )
)
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)

Then flush at the end of predict, before returning. Model Serving keeps the container alive between requests, so the flush is what sends spans promptly rather than leaving them queued:

    def predict(self, request: ResponsesAgentRequest) -> ResponsesAgentResponse:
        response = ...  # existing agent logic, unchanged
        tracer_provider.force_flush(15000)
        return response

Declare the dependencies

The serving environment installs only the packages declared on the model, so add these to the existing pip_requirements on log_model. Without them the agent fails to import at request time:

        pip_requirements=[
            # existing requirements
            "opentelemetry-sdk",
            "opentelemetry-exporter-otlp-proto-http",
            "openinference-instrumentation-openai",
        ],

Pass the endpoint and key

Add these to environment_vars on agents.deploy(). OTEL_SERVICE_NAME is optional and sets the name used to filter traces later:

    environment_vars={
        "ORQ_OTEL_ENDPOINT": "https://api.orq.ai/v2/otel/v1/traces",
        "ORQ_API_KEY": "{{secrets/orq/api_key}}",
        "OTEL_SERVICE_NAME": "databricks-support-agent",
    },

Configuring OTLP through the standard OTEL_EXPORTER_OTLP_* variables alone does not export traces from Model Serving. The managed runtime sets up its own MLflow tracing, which takes precedence, and spans reach the Databricks experiment rather than Orq.ai. Add the exporter to the agent as shown above instead.

A secret reference is substituted only when it is the entire value of a variable. Writing Authorization=Bearer {{secrets/orq/api_key}} sends the braces literally and Orq.ai rejects the spans with a 401. Pass the key in its own variable, as above, and build the header in the agent.

View traces

Traces appear in AI Studio under the Traces tab. Each LLM call arrives as a span carrying the request and response messages, prompt and completion tokens, the resolved model and provider, and computed cost. Filter on the value set in OTEL_SERVICE_NAME to isolate one agent.

Complete example

This example calls a Databricks native endpoint. To route the agent through the AI Gateway instead, set LLM_ENDPOINT to the endpoint created above.

Full agent module and deployment script

import os

from databricks.sdk import WorkspaceClient
from mlflow.models import set_model
from mlflow.pyfunc import ResponsesAgent
from mlflow.types.responses import (
    ResponsesAgentRequest,
    ResponsesAgentResponse,
    create_text_output_item,
)
from openinference.instrumentation.openai import OpenAIInstrumentor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

LLM_ENDPOINT = "databricks-gemini-3-5-flash"

tracer_provider = TracerProvider()
tracer_provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(
            endpoint=os.environ["ORQ_OTEL_ENDPOINT"],
            headers={"Authorization": f"Bearer {os.environ['ORQ_API_KEY']}"},
        )
    )
)
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)


class SupportAgent(ResponsesAgent):
    def predict(self, request: ResponsesAgentRequest) -> ResponsesAgentResponse:
        client = WorkspaceClient().serving_endpoints.get_open_ai_client()
        completion = client.chat.completions.create(
            model=LLM_ENDPOINT,
            messages=[{"role": "user", "content": self._question(request)}],
            max_tokens=300,
        )
        tracer_provider.force_flush(15000)
        return ResponsesAgentResponse(
            output=[
                create_text_output_item(
                    text=completion.choices[0].message.content, id="answer"
                )
            ]
        )

    def _question(self, request: ResponsesAgentRequest) -> str:
        last_message = request.input[-1]
        as_dict = (
            last_message
            if isinstance(last_message, dict)
            else last_message.model_dump()
        )
        return as_dict.get("content", "")


set_model(SupportAgent())

import mlflow
from databricks import agents
from mlflow.models.resources import DatabricksServingEndpoint

mlflow.set_registry_uri("databricks-uc")

with mlflow.start_run():
    logged_agent = mlflow.pyfunc.log_model(
        name="agent",
        python_model="agent.py",
        resources=[
            DatabricksServingEndpoint(endpoint_name="databricks-gemini-3-5-flash")
        ],
        pip_requirements=[
            "mlflow>=3.11",
            "databricks-sdk",
            "openai",
            "httpx",
            "opentelemetry-sdk",
            "opentelemetry-exporter-otlp-proto-http",
            "openinference-instrumentation-openai",
        ],
    )

registered_agent = mlflow.register_model(
    logged_agent.model_uri, "catalog.schema.support_agent"
)

agents.deploy(
    "catalog.schema.support_agent",
    registered_agent.version,
    environment_vars={
        "ORQ_OTEL_ENDPOINT": "https://api.orq.ai/v2/otel/v1/traces",
        "ORQ_API_KEY": "{{secrets/orq/api_key}}",
        "OTEL_SERVICE_NAME": "databricks-support-agent",
    },
)

Evaluations & Experiments

Once agents are running, use Evaluatorq to score outputs across a dataset and Experiments to compare configurations side by side.

Run Evaluations with Evaluatorq

Run parallel evaluations across agents and compare results.

Run Experiments via the API

Compare agent configurations and view results in the AI Studio.

Model Providers

Coding Agents

Chat Interfaces

Frameworks & SDKs

Automations

AI Gateway

Observability

Prerequisites

Store the API key

AI Gateway

Observability

Add the exporter to the agent

Declare the dependencies

Pass the endpoint and key

View traces

Complete example

Evaluations & Experiments

Run Evaluations with Evaluatorq

Run Experiments via the API

AI Gateway

Observability

​Prerequisites

​Store the API key

​AI Gateway

​Observability

​Add the exporter to the agent

​Declare the dependencies

​Pass the endpoint and key

​View traces

​Complete example

​Evaluations & Experiments

Run Evaluations with Evaluatorq

Run Experiments via the API

Prerequisites

Store the API key

AI Gateway

Observability

Add the exporter to the agent

Declare the dependencies

Pass the endpoint and key

View traces

Complete example

Evaluations & Experiments