Skip to main content

​
Use cases

  • Find recurring user intents, sentiment patterns, and failures across production conversations.
  • Prioritize improvements with failure rates, satisfaction signals, and trace evidence.

​
Overview

Insights groups production conversations into named clusters, measures their quality, and produces a report with failure patterns and recommended actions. The analysis window selects matching conversations. For each selected conversation, Insights analyzes the full thread as one transcript, including traces outside the window, and falls back to a single trace when no thread is present.

​
Run an analysis

1

Open Insights

Open AI Observability, then select Insights.
2

Configure the analysis

Set the analysis window, sampling percentage, filters, profile, schedule, and conversation pass mode. Save the configuration before starting a run.
3

Start the run

Select Run analysis. Monitor the run as it moves from Queued to Running, then to Completed, Failed, or Canceled.
4

Review the report

Open a completed run to inspect clusters, report artifacts, quality measurements, and recommended actions.
Cancel a queued or running analysis from the run view. A cancellation request stops further processing and marks the run as Canceled.

​
Configure Insights

Set a maximum conversation count and cost ceiling to bound each run. Insights accepts up to 5,000 conversations per run.

​
Interpret the results

​
Dimensions

Review clusters across these dimensions:
  • Intent: groups conversations by the request being made.
  • Positive sentiment, Negative sentiment, and Neutral sentiment: group conversations by detected sentiment.
  • Failure: groups conversations by assistant errors.
Each cluster includes a name, description, conversation count, and supporting conversations. Open a conversation to inspect its summary, assignment confidence, annotation, and related traces.

​
Report

Use the report to move from discovered patterns to an improvement plan:
  • Failure taxonomy: review failure modes with their counts, rates, and classifications.
  • Quadrants: compare cluster volume with satisfaction and failure rate. Results appear in Danger Zone, Maintain, High ROI, or Low ROI.
  • Action plan: work through P0, P1, and P2 recommendations with hypotheses, success criteria, affected clusters, and evidence traces.
  • Summary statistics: review conversation and cluster counts, pass rate, satisfaction, frustration, and concerning conversation counts.

​
Quality

Use structural quality measurements to compare clustering results. Enable the optional LLM judge to add intra-cluster coherence, inter-cluster separation, and composite scores. Re-evaluate a completed run with a different judge model or version without replacing earlier quality results.

​
Continuous assignment

Enable a schedule to run incremental analysis from the last successful window. After a successful run creates centroids, Insights assigns new completed conversations to the nearest clusters in scheduled micro-batches. Continuous assignment uses the run’s pinned embedding model and does not call an LLM for cluster matching.

​
Availability and cost

Insights is available on all Orq.ai SaaS plans. For on-premise installations, enable the Insights license flag and the workspace feature. A workspace kill switch can disable the feature in either deployment mode. Insights uses model generation and embedding calls during analysis. SaaS usage is charged to the workspace through the standard credits and billing flow. On-premise usage uses configured provider integrations. Set sampling, the conversation limit, pass mode, optional judge, and cost ceiling according to the available budget. Public API and SDK access are not available for Insights.