Test a retrieval-augmented generation (RAG) system in two stages: first check whether the retriever finds the right evidence, then check whether the generator answers accurately from that evidence. Add end-to-end regression tests to catch failures across the full pipeline. A useful evaluation set, metric-specific pass thresholds, trace-level debugging, and human review for high-risk cases matter more than a single headline score.
What does “RAG accuracy” mean?
RAG accuracy is not one property. A system can retrieve the right passages and still produce an unsupported answer; it can also write a convincing answer from incomplete or irrelevant evidence. Evaluate retrieval and generation separately before judging the combined result.
- Retrieval: Did the system find relevant evidence, and did it rank that evidence high enough to reach the generator?
- Generation: Does the answer address the question, and are its claims supported by the evidence supplied to the model?
- End to end: Does the final response meet the requirements for a real user question, including correctness, completeness, and any domain-specific constraints?
These dimensions should remain visible in reports. A single aggregate score can conceal a retrieval regression behind stronger answer wording, or hide failures in a small but important category of questions.
Build an evaluation set that reflects actual use
Collect representative and difficult questions
Start with real user questions, production failure reports, and support tickets. Add deliberately difficult cases: ambiguous wording, questions that require evidence from multiple chunks, questions whose answer is absent from the corpus, and cases where similar documents could be confused. Include high-risk or domain-specific scenarios if the system will handle them.
#1 Best Overall
For each item, record the question and, where practical, reference claims or an expected answer, acceptable evidence IDs, and the relevant document or chunk versions. Preserve the question as it was asked; rewriting every item into a clean, unambiguous form can make the set less representative.
Separate development, regression, and held-out data
- Development set: Use this to diagnose failures and tune chunking, retrieval, prompts, or models.
- Regression set: Keep a stable, versioned core that runs on each change and supports comparison with an accepted baseline.
- Held-out set: Reserve examples from routine tuning so you can check whether apparent gains generalize beyond the cases used during iteration.
Watch for leakage between document updates and evaluation labels. If an answer key was written from a newer document version than the one used in a test run, the test may measure a version mismatch rather than system quality. Track which corpus and label versions apply to each example.
Log enough information to reproduce a result
Alongside the test inputs and labels, store the retrieved chunks, retriever configuration, prompt and model versions, evaluator configuration, latency, token cost, and evaluator outputs. Version the dataset and test configuration. Without that context, a score change may be impossible to attribute to the retriever, ingestion, chunking, prompt, generation model, or judge.
Test retrieval before generation
Run the retriever against questions with labeled relevant evidence. This isolates whether useful context is available to the answer model and helps distinguish retrieval failures from generation failures.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →| Metric | What it tells you | What to inspect |
|---|---|---|
| Context recall | How much of the labeled relevant evidence was retrieved. | Relevant evidence that was missed, especially when it is necessary to answer correctly. |
| Context precision | How much of the retrieved context is relevant to the question. | Irrelevant or distracting chunks that consume the context window or introduce competing claims. |
| Reciprocal rank or average precision | Whether relevant evidence appears near the top of the ranked results. | Cases where relevant evidence was retrieved but ranked too low to be useful downstream. |
Ragas’ official metric catalog includes context precision and context recall. RagaAI’s framework also describes deterministic, rank-aware, and LLM-based ways to assess context. The exact definition and implementation of a metric can differ, so document which evaluator and settings produced each score.
Retrieval scores depend on the relevance labels. Define what counts as acceptable evidence for each question, including whether one sufficient chunk is enough or several pieces are required. If a question has multiple valid sources, label those alternatives rather than treating only one document as correct.
Test whether answers use evidence correctly
Once retrieval is measured on its own, evaluate answers using the actual context returned for each question. Include both questions with reliable reference answers and questions where the retrieved context is insufficient to support a definitive response.
| Metric or check | Question it answers | Best use |
|---|---|---|
| Faithfulness | Are the answer’s claims supported by the supplied context? | Finding unsupported claims, contradictions, or details the context does not establish. |
| Response relevancy | Does the answer address the user’s question? | Detecting answers that are off-topic, evasive, or focused on the wrong part of a question. |
| Reference-based factual correctness | Does the answer agree with reliable expected facts or reference claims? | Questions with a trustworthy answer key or clearly defined required claims. |
| Exact match | Does the output match an expected string? | Narrow tasks where wording or format is itself the requirement, not open-ended answers. |
Ragas lists faithfulness and response relevancy among its metrics, alongside context entities recall, noise sensitivity, and multimodal variants. These tools can help assess distinct aspects of a RAG response, but no single metric establishes that an answer is true in the real world. Reference-based checks are only as dependable as the reference; context-based checks can confirm consistency with context that is itself outdated or wrong.
Keep the dimensions separate while diagnosing results. If an answer is unfaithful, inspect whether the retrieved context contained the right evidence. If it was faithful but incorrect, investigate the source documents, their freshness, and the labels. If it was supported but irrelevant, inspect the question interpretation and response behavior.
Can an LLM judge replace human review?
An LLM judge can scale evaluation, but it should not be treated as a universal substitute for human assessment. The RAGAS authors’ EACL 2024 paper presents metrics intended to evaluate different dimensions without requiring ground-truth human annotations. That reduces the need to label every example, but metric outputs still need calibration and interpretation.
NIST’s 2025 study of TREC 2024 RAG relevance assessment examined 77 runs from 19 teams. It reported that UMBRELA-generated relevance assessments correlated highly with manual rankings. That is evidence for a specific assessor and benchmark—not proof that any LLM judge will agree with experts on any domain or task.
Make judge results auditable
- Write a rubric with explicit pass/fail criteria for the dimension being judged, such as whether each material answer claim has support.
- Require the judge to identify the supporting context span, rather than returning only a score or verdict.
- When comparing candidate systems, randomize or blind answer order to reduce position effects.
- Periodically compare judge decisions with human labels and review disagreements, especially around borderline cases.
- Retain human review for high-risk, novel, or domain-specific questions where an incorrect answer could cause harm.
Use deterministic checks where they fit—for example, required fields or exact output formats—and judges for evaluations that require semantic interpretation. Keep judge model, rubric, and configuration versions in the run record; changing the evaluator can change scores even if the RAG system itself is unchanged.
Recommended Free Tools
Put RAG evaluation in a CI and monitoring loop
- Freeze the test inputs and configuration. Record the dataset version, retriever settings, prompt, model, and evaluator configuration for the run.
- Run retrieval and generation tests on each change. Report their results separately, then include end-to-end checks for final responses.
- Compare with the last accepted baseline. Set tolerances for each metric rather than relying on an overall score. The appropriate threshold depends on the system’s use and risk; the cited sources do not prescribe one universal cutoff.
- Gate critical regressions. Fail the build or require review when an important slice regresses, even if the aggregate score improves.
- Save traces and evaluator explanations. Use them to identify whether a failure originated in ingestion, chunking, retrieval, prompting, generation, or judging.
- Refresh carefully. Add new production questions and human-reviewed failures periodically, while preserving the stable regression core so results remain comparable over time.
LangChain’s documented workflow combines Ragas metrics with LangSmith traces and datasets for continuous evaluation, including adding examples from human feedback. OpenAI’s optimization guidance recommends automated evaluation with explicit scorecards to speed iteration and discusses RAG as a technique for improving accuracy and consistency. These are implementation patterns; select tools based on your data handling and operational requirements.
Choose an evaluation approach for your constraints
Before adopting a framework, compare it against the needs of your application rather than treating a feature list as a quality ranking.
- Evidence labels: Can you label relevant chunks or reference claims, and how much manual labeling can you maintain?
- Coverage: Does the approach test retrieval, generation, or both?
- Scoring: Which checks are deterministic, and which rely on an LLM judge? How will you calibrate and reproduce judge results?
- Operations: Does it support dataset versioning, trace-level debugging, and the CI workflow you need?
- Constraints: Does it meet your latency, cost, privacy, data-residency, language, modality, and domain requirements?
Ragas is a direct fit when you need RAG-specific metrics. LangSmith can support trace, dataset, and continuous-regression workflows in the documented LangChain example. OpenAI’s guidance is useful for scorecard-based automated evaluation and RAG optimization. Tool choice does not remove the need to define labels, choose thresholds, and review failures in context.
How to interpret a score without overclaiming
Use scores as signals for finding and tracking failures, not as proof of real-world truth. Retrieval metrics depend on how relevance is defined; generation metrics depend on the context, references, rubric, and evaluator. A rise in one metric can coexist with a serious regression in a smaller category.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
Review errors by slice—for example, question type, source collection, language, or risk level—and compare the same versioned set against a frozen baseline. If you change the dataset, prompt, model, retriever, or evaluator, record that change so readers of the scorecard know what the comparison actually measures. For regulated or safety-critical deployments, retain domain-specific acceptance tests and human review rather than relying on automated metrics alone.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
