Three new RAGAS Evaluators for enhanced RAG Experiments
We’ve added three new RAGAS evaluators to the Orq.ai platform, giving you deeper insights into your retrieval-augmented generation (RAG) workflows. These evaluators help you understand not just if the right information was retrieved, but how well your system handles noisy data and captures key context.
Here’s what’s new:
1. Context Recall
Measure how much of the relevant reference information your retrieval pipeline actually brings into context. This evaluator compares the retrieved text to the reference for each user query, helping you identify if important facts are missing. Use context recall to improve your retrieval strategy and ensure your LLM responses are always well-supported.
Entities used:
- Reference (ground truth)
- Retrieved text
When to use:
- When you want to know if your retrieval step is actually surfacing all necessary information for the LLM to answer correctly.
2. Noise Sensitivity
Understand how your RAG system performs when irrelevant or “noisy” information is mixed into the retrieved context. This evaluator tests the robustness of your model by introducing noise and measuring the impact on your outputs. It’s a great way to benchmark reliability in real-world scenarios where retrieval isn’t always perfect.
Entities used:
- User message
- Reference (ground truth)
- Retrieved text (with added noise)
When to use:
- When you want to see if your LLM can still provide accurate answers, even when extra, unrelated information appears in the context.
3. Context Entities Recall
Check whether the most important entities (names, places, organizations, etc.) present in your reference are also present in the retrieved context. This evaluator goes beyond text overlap and focuses on the coverage of key information units, helping you catch subtle gaps in your retrieval pipeline.
Entities used:
- Reference (ground truth)
- Retrieved text
When to use:
- When you want to be sure your retrieval is capturing and surfacing all the essential entities needed to answer user queries correctly.
Use these new RAGAS evaluators to systematically assess and improve the quality, completeness, and reliability of your RAG pipelines right from within Orq.ai.