SmolAgents

Integrate Orq.ai with SmolAgents (Hugging Face) using OpenTelemetry and MLflow

Getting Started

SmolAgents is Hugging Face's lightweight agent framework for building AI agents with tool calling capabilities. Tracing SmolAgents with Orq.ai provides comprehensive insights into agent reasoning, tool usage, code execution, and performance to optimize your agent workflows.

Prerequisites

Before you begin, ensure you have:

  • An Orq.ai account and API key
  • Python 3.8+ installed
  • SmolAgents installed in your project
  • API keys for your LLM providers (OpenAI, Anthropic, etc.)

Install Dependencies

# Core SmolAgents
pip install smolagents

# OpenTelemetry packages
pip install opentelemetry-sdk opentelemetry-exporter-otlp

# MLflow for advanced tracking
pip install mlflow

# LLM providers (choose what you need)
pip install openai anthropic transformers

Configure Orq.ai

Set up your environment variables to connect to Orq.ai's OpenTelemetry collector:

Unix/Linux/macOS:

export OTEL_EXPORTER_OTLP_ENDPOINT="https://api.orq.ai/v2/otel"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer <ORQ_API_KEY>"
export OTEL_RESOURCE_ATTRIBUTES="service.name=smolagents-app,service.version=1.0.0"
export OPENAI_API_KEY="<YOUR_OPENAI_API_KEY>"

Windows (PowerShell):

$env:OTEL_EXPORTER_OTLP_ENDPOINT = "https://api.orq.ai/v2/otel"
$env:OTEL_EXPORTER_OTLP_HEADERS = "Authorization=Bearer <ORQ_API_KEY>"
$env:OTEL_RESOURCE_ATTRIBUTES = "service.name=smolagents-app,service.version=1.0.0"
$env:OPENAI_API_KEY = "<YOUR_OPENAI_API_KEY>"

Using .env file:

OTEL_EXPORTER_OTLP_ENDPOINT=https://api.orq.ai/v2/otel
OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer <ORQ_API_KEY>
OTEL_RESOURCE_ATTRIBUTES=service.name=smolagents-app,service.version=1.0.0
OPENAI_API_KEY=<YOUR_OPENAI_API_KEY>

Integrations

SmolAgents supports multiple integration approaches for comprehensive observability:

MLflow Integration

Set up MLflow tracking:

# smolagents_mlflow.py
import mlflow
import mlflow.openai
from smolagents import CodeAgent, ToolCallingAgent
from smolagents.tools import PythonInterpreterTool, DuckDuckGoSearchTool

# Configure MLflow
mlflow.set_tracking_uri("https://api.orq.ai/v2/mlflow")
mlflow.set_experiment("smolagents-experiment")

# Enable autologging
mlflow.openai.autolog()

def create_agent_with_mlflow():
    with mlflow.start_run():
        # Log parameters
        mlflow.log_param("agent_type", "CodeAgent")
        mlflow.log_param("model", "gpt-4")

        # Create and configure agent
        agent = CodeAgent(
            tools=[PythonInterpreterTool(), DuckDuckGoSearchTool()],
            model="gpt-4",
            max_steps=10
        )

        # Log agent configuration
        mlflow.log_param("max_steps", agent.max_steps)
        mlflow.log_param("tool_count", len(agent.tools))

        return agent

def run_agent_task(agent, task):
    with mlflow.start_run(nested=True, run_name="agent_task"):
        mlflow.log_param("task", task)

        start_time = time.time()
        result = agent.run(task)
        duration = time.time() - start_time

        # Log metrics
        mlflow.log_metric("execution_time", duration)
        mlflow.log_metric("steps_taken", len(result.steps) if hasattr(result, 'steps') else 0)

        # Log the result
        mlflow.log_text(result, "agent_output.txt")

        return result

Manual OpenTelemetry Setup

Create comprehensive tracing configuration:

# tracing.py
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import Resource
from opentelemetry.semconv.resource import ResourceAttributes
from opentelemetry.propagate import set_global_textmap
from opentelemetry.propagate.textmap import CompositePropagator
from opentelemetry.propagate.b3 import B3MultiFormat
from opentelemetry.propagate.baggage import BaggagePropagator
import os

# Configure resource
resource = Resource.create({
    ResourceAttributes.SERVICE_NAME: "smolagents-app",
    ResourceAttributes.SERVICE_VERSION: "1.0.0",
    ResourceAttributes.DEPLOYMENT_ENVIRONMENT: os.getenv("ENVIRONMENT", "development"),
})

# Configure exporter
exporter = OTLPSpanExporter(
    endpoint=os.getenv("OTEL_EXPORTER_OTLP_ENDPOINT") + "/v1/traces",
    headers={"Authorization": os.getenv("OTEL_EXPORTER_OTLP_HEADERS").split("=")[1]}
)

# Set up tracing
provider = TracerProvider(resource=resource)
processor = BatchSpanProcessor(exporter)
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

# Configure propagation
set_global_textmap(CompositePropagator([
    B3MultiFormat(),
    BaggagePropagator()
]))

# Get tracer
tracer = trace.get_tracer("smolagents", "1.0.0")

Examples

Basic Code Agent with Tracing

import time
from smolagents import CodeAgent
from smolagents.tools import PythonInterpreterTool
from tracing import tracer
from opentelemetry import trace, baggage
from opentelemetry.semconv.trace import SpanAttributes

async def basic_code_agent_example():
    # Set baggage for context propagation
    ctx = baggage.set_baggage("user.id", "user123")

    with tracer.start_as_current_span(
        "smolagents.code_agent.run",
        context=ctx,
        attributes={
            "agent.type": "CodeAgent",
            "agent.model": "gpt-4",
            "agent.max_steps": 5,
        }
    ) as span:
        try:
            # Create agent
            agent = CodeAgent(
                tools=[PythonInterpreterTool()],
                model="gpt-4",
                max_steps=5
            )

            task = "Calculate the fibonacci sequence up to 20 and plot it"
            span.set_attribute("agent.task", task)

            start_time = time.time()
            result = agent.run(task)
            duration = time.time() - start_time

            # Add metrics
            span.set_attributes({
                "agent.execution_time": duration,
                "agent.result.length": len(str(result)),
                "agent.tools.count": len(agent.tools),
            })

            print("Agent Result:", result)
            return result

        except Exception as e:
            span.record_exception(e)
            span.set_status(trace.StatusCode.ERROR, str(e))
            raise

# Run example
import asyncio
asyncio.run(basic_code_agent_example())

Tool Calling Agent with Custom Tools

from smolagents import ToolCallingAgent, Tool
from smolagents.tools import DuckDuckGoSearchTool
from tracing import tracer
import requests
import json

class WeatherTool(Tool):
    name = "get_weather"
    description = "Get current weather for a location"

    def forward(self, location: str) -> str:
        with tracer.start_as_current_span(
            "tool.weather.execute",
            attributes={
                "tool.name": self.name,
                "tool.location": location,
            }
        ) as span:
            try:
                # Mock weather API call
                weather_data = {
                    "location": location,
                    "temperature": "22°C",
                    "condition": "sunny",
                    "humidity": "65%"
                }

                result = f"Weather in {location}: {weather_data['temperature']}, {weather_data['condition']}"

                span.set_attributes({
                    "tool.result.temperature": weather_data["temperature"],
                    "tool.result.condition": weather_data["condition"],
                    "tool.result.humidity": weather_data["humidity"],
                })

                return result

            except Exception as e:
                span.record_exception(e)
                span.set_status(trace.StatusCode.ERROR, str(e))
                raise

class CalculatorTool(Tool):
    name = "calculator"
    description = "Perform mathematical calculations"

    def forward(self, expression: str) -> str:
        with tracer.start_as_current_span(
            "tool.calculator.execute",
            attributes={
                "tool.name": self.name,
                "tool.expression": expression,
            }
        ) as span:
            try:
                # Safe evaluation for basic math
                allowed_chars = set('0123456789+-*/.() ')
                if not all(c in allowed_chars for c in expression):
                    raise ValueError("Invalid characters in expression")

                result = eval(expression)  # Note: In production, use a safer math parser

                span.set_attributes({
                    "tool.result.value": result,
                    "tool.expression.safe": True,
                })

                return f"Result: {result}"

            except Exception as e:
                span.record_exception(e)
                span.set_status(trace.StatusCode.ERROR, str(e))
                raise

async def tool_calling_agent_example():
    with tracer.start_as_current_span(
        "smolagents.tool_calling_agent.run",
        attributes={
            "agent.type": "ToolCallingAgent",
            "agent.model": "gpt-4",
        }
    ) as span:
        # Create agent with custom tools
        agent = ToolCallingAgent(
            tools=[
                WeatherTool(),
                CalculatorTool(),
                DuckDuckGoSearchTool()
            ],
            model="gpt-4"
        )

        span.set_attribute("agent.tools.count", len(agent.tools))
        span.set_attribute("agent.tools.names", [tool.name for tool in agent.tools])

        task = "What's the weather in Paris? Then calculate 15% tip on a $50 bill."
        span.set_attribute("agent.task", task)

        start_time = time.time()
        result = agent.run(task)
        duration = time.time() - start_time

        span.set_attributes({
            "agent.execution_time": duration,
            "agent.result.length": len(str(result)),
        })

        print("Agent Result:", result)
        return result

# Run example
import asyncio
asyncio.run(tool_calling_agent_example())

Multi-Step Agent Workflow with Detailed Tracing

from smolagents import CodeAgent
from smolagents.tools import PythonInterpreterTool, DuckDuckGoSearchTool
from tracing import tracer
import time

class TracedAgent:
    def __init__(self, agent_type="CodeAgent", model="gpt-4"):
        self.agent_type = agent_type
        self.model = model
        self.tools = [PythonInterpreterTool(), DuckDuckGoSearchTool()]

    def create_agent(self):
        with tracer.start_as_current_span(
            "smolagents.agent.create",
            attributes={
                "agent.type": self.agent_type,
                "agent.model": self.model,
                "agent.tools.count": len(self.tools),
            }
        ) as span:
            agent = CodeAgent(
                tools=self.tools,
                model=self.model,
                max_steps=10
            )

            span.set_attributes({
                "agent.max_steps": agent.max_steps,
                "agent.tools.names": [tool.__class__.__name__ for tool in self.tools],
            })

            return agent

    def execute_task(self, agent, task, step_name):
        with tracer.start_as_current_span(
            f"smolagents.task.{step_name}",
            attributes={
                "task.name": step_name,
                "task.description": task,
                "agent.type": self.agent_type,
            }
        ) as span:
            try:
                start_time = time.time()
                result = agent.run(task)
                duration = time.time() - start_time

                span.set_attributes({
                    "task.execution_time": duration,
                    "task.result.length": len(str(result)),
                    "task.status": "completed",
                })

                return result

            except Exception as e:
                span.record_exception(e)
                span.set_status(trace.StatusCode.ERROR, str(e))
                span.set_attribute("task.status", "failed")
                raise

async def multi_step_workflow_example():
    with tracer.start_as_current_span(
        "smolagents.workflow.data_analysis",
        attributes={
            "workflow.name": "data_analysis",
            "workflow.steps.count": 3,
        }
    ) as workflow_span:
        traced_agent = TracedAgent()
        agent = traced_agent.create_agent()

        tasks = [
            ("research", "Search for recent data on climate change trends"),
            ("analysis", "Create a Python script to analyze temperature data from the last 10 years"),
            ("visualization", "Generate a graph showing the temperature trends and save it as an image")
        ]

        workflow_span.set_attribute("workflow.tasks", [task[0] for task in tasks])

        results = []
        for step_name, task in tasks:
            result = traced_agent.execute_task(agent, task, step_name)
            results.append((step_name, result))

            # Add delay between steps for demonstration
            time.sleep(1)

        workflow_span.set_attributes({
            "workflow.status": "completed",
            "workflow.results.count": len(results),
        })

        return results

# Run workflow example
import asyncio
results = asyncio.run(multi_step_workflow_example())
for step_name, result in results:
    print(f"Step '{step_name}' completed: {str(result)[:100]}...")

Error Handling and Recovery

from smolagents import CodeAgent
from smolagents.tools import PythonInterpreterTool
from tracing import tracer
from opentelemetry.semconv.trace import SpanAttributes

async def error_handling_example():
    with tracer.start_as_current_span(
        "smolagents.error_handling.demo",
        attributes={
            "demo.type": "error_recovery",
        }
    ) as span:
        agent = CodeAgent(
            tools=[PythonInterpreterTool()],
            model="gpt-4",
            max_steps=3
        )

        # Test scenarios with potential failures
        test_cases = [
            ("valid_task", "Calculate 2 + 2"),
            ("invalid_syntax", "Run this broken Python: print('hello' + 123 +)"),
            ("resource_intensive", "Create an infinite loop and see what happens"),
        ]

        results = []

        for test_name, task in test_cases:
            with tracer.start_as_current_span(
                f"smolagents.test_case.{test_name}",
                attributes={
                    "test.name": test_name,
                    "test.task": task,
                }
            ) as test_span:
                try:
                    start_time = time.time()
                    result = agent.run(task)
                    duration = time.time() - start_time

                    test_span.set_attributes({
                        "test.status": "success",
                        "test.execution_time": duration,
                        "test.result.length": len(str(result)),
                    })

                    results.append((test_name, "success", result))

                except Exception as e:
                    test_span.record_exception(e)
                    test_span.set_status(trace.StatusCode.ERROR, str(e))
                    test_span.set_attributes({
                        "test.status": "failed",
                        "test.error.type": type(e).__name__,
                        "test.error.message": str(e),
                    })

                    results.append((test_name, "failed", str(e)))

        # Summary
        successful_tests = sum(1 for _, status, _ in results if status == "success")
        failed_tests = len(results) - successful_tests

        span.set_attributes({
            "demo.total_tests": len(results),
            "demo.successful_tests": successful_tests,
            "demo.failed_tests": failed_tests,
            "demo.success_rate": successful_tests / len(results) if results else 0,
        })

        return results

# Run error handling example
import asyncio
results = asyncio.run(error_handling_example())
for test_name, status, result in results:
    print(f"Test '{test_name}': {status} - {str(result)[:100]}...")

Next Steps

Verify traces: Check your Orq.ai dashboard to see incoming traces ✅ Add custom attributes: Enhance traces with business-specific metadata ✅ Set up alerts: Configure monitoring for agent performance degradation ✅ Explore metrics: Use trace data for agent optimization ✅ Monitor tool usage: Track which tools are used most frequently

Related Documentation

Support