iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
If a RAG agent retrieves the wrong chunk, trace the passage through the pipeline before changing the prompt or embedding model. The mismatch may start in ingestion or chunking, appear when query language differs from source wording, or happen later when filters, ranking, or context limits discard a useful result. Log each stage and test fixes against representative questions; these five failure modes are diagnostic categories, not a universal ranking.
First find where the expected passage disappears
For each test question, identify the source passage that should support the answer. Then trace whether that passage was indexed, retrieved as a candidate, kept after ranking and filtering, and included in the final context sent to the model. This separates a retrieval miss from a context-assembly problem or a generation failure.
- Build a small, versioned evaluation set. Use real user questions and record the expected source passage for each. Keep the set stable while comparing pipeline changes.
- Capture one end-to-end trace per question. Log the query after preprocessing; candidate IDs, text, and scores; active filters and ranking steps; and the final ordered chunks passed to the model.
- Locate the first stage that loses the passage. If it is absent from candidates, inspect the indexed source, chunking, preprocessing, embeddings, and search mode. If it is a candidate but absent from final context, inspect filters, reranking, result limits, and context assembly.
- Change one stage at a time. Compare retrieval relevance, answer quality, and latency on the same evaluation set. Microsoft’s architecture guidance recommends evaluating each search step; retrieval and reranking settings involve trade-offs rather than a single universally best configuration.
When the expected passage reaches the model but does not support a grounded answer, investigate whether the chunk is complete and whether generation is instructed to use retrieved evidence appropriately. Retrieval quality and prompt design both affect RAG quality, as Microsoft Foundry documentation notes.
1. Does the chunk preserve the context needed to answer?
A search result can be topically relevant yet unusable if its boundaries cut away the heading, qualification, definition, or neighboring sentence that makes the passage meaningful. The reverse problem occurs when a chunk combines unrelated sections: its vector may represent several ideas at once, making it a poor match for any one question.
Inspect boundaries and coherence
- Check whether headings and their explanatory text land together, and whether a passage begins or ends in the middle of a definition, list, or qualification.
- Look for chunks that join unrelated topics or repeat so much overlapping text that retrieval returns near-duplicates instead of distinct evidence.
- Check whether the expected answer depends on adjacent content that was split into another chunk. If so, consider a structure-aware boundary or a retrieval design that can include neighboring chunks.
Choose chunking by corpus and questions
Fixed-size chunking is straightforward to implement. Structure-aware or semantic approaches aim to preserve coherent document units, but need to be evaluated against the actual corpus and question set. Microsoft Learn gives 512 tokens with 25% overlap—128 tokens—as an initial chunk-size and overlap recommendation in its guidance. Treat that as a vendor starting point, not an optimum for every document collection. The same source reports an 8,191-token maximum input length for Azure OpenAI text-embedding-3-small and estimates around 6,000 words for common OpenAI models; that limit is model-specific, not a recommended chunk size, and should be rechecked against current model documentation.
2. Are the question and source speaking different languages?
A user may ask conversationally while the source uses formal terminology, product names, abbreviations, or an exact identifier. Vector retrieval can help with semantic similarity; keyword retrieval can help with exact terms. Hybrid search combines the two, and semantic ranking can reorder results by meaning. Which works best depends on the questions and content, so compare them on the same evaluation set rather than assuming vector search alone is sufficient.
Test search modes against the query set
- Compare keyword, vector, and hybrid retrieval, checking whether exact terminology and identifiers are found as well as paraphrased questions.
- For queries with several parts, test decomposition into focused subqueries; for ambiguous phrasing, test rewriting or augmentation.
- HyDE, or hypothetical document embeddings, is another query-translation approach to evaluate, not a guaranteed improvement.
- Record changes in whether the expected passage appears among candidates and where it ranks. A better-looking top result for one question is not enough to establish an overall improvement.
Microsoft’s Azure AI Search guidance describes hybrid retrieval and semantic ranking; Microsoft’s broader RAG guidance also discusses query translation approaches such as augmentation, decomposition, rewriting, and HyDE. These are implementation options, not evidence that one mode wins for every workload.
3. Are preprocessing and embeddings compatible?
Similarity scores are only useful when the indexed text and query are represented consistently enough for the task. If indexing and query-time preprocessing differ, important terms can be removed, normalized differently, or interpreted inconsistently. An embedding change can also leave an index encoded by a different model or configuration than the one used to process new queries.
Compare the actual inputs
- Inspect the stored chunk text and the query after preprocessing, not only the original document and user question.
- Verify that the embedding model and relevant configuration used for queries are compatible with those used to build the index. If either changed, confirm whether the indexed vectors were rebuilt as needed.
- Preserve negation and other meaning-bearing words. Microsoft’s guidance specifically cautions that “not” can carry important meaning; removing it can reverse a query’s intent.
- Check whether cleaning removed punctuation, identifiers, units, or formatting that distinguishes one passage from another.
Do not diagnose a poor match from a score alone. Inspect the query and candidate text, then compare retrieval behavior against expected passages in the evaluation set.
4. Are limits, filters, or ranking hiding a good candidate?
A passage can be retrieved early and still never reach the model. A restrictive filter may exclude it; a small candidate limit may cut it off; reranking may move it below the final cutoff; or context assembly may omit it after ranking. Retrieval generally needs to find relevant candidates, while reranking can improve precision by evaluating query-to-chunk relevance. Reranking adds processing latency, so measure its value in the actual workload.
Follow the candidate through each cutoff
- Log candidate recall at the configured retrieval limit: does the expected passage appear before reranking?
- Check every filter, including metadata and access-control filters, to confirm that the document is eligible for that query.
- Compare the candidate pool size with the number of chunks ultimately passed to the model. A larger pool may improve the chance of finding evidence but can add latency, token cost, and irrelevant material to later stages.
- Inspect reranker inputs and outputs, the final top-N limit, and the order used by context assembly.
- Verify that the expected result is not dropped by field selection, deduplication, or a context-window budget.
Compare retrieval with and without reranking, and compare a larger candidate pool with a smaller final context. Measure passage coverage, answer quality, and latency together; more candidates are not automatically better if they create noise or exceed the context budget.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
5. Is the indexed content fresh, complete, and correctly enriched?
If the expected passage never appears in candidates, confirm that it exists in the searchable index before tuning ranking. The source may not have been ingested, a recent update may not have been persisted, or the passage may have been split or transformed incorrectly. Wrong document identity or metadata can also make a valid passage unreachable through the query’s filters.
Audit ingestion stage by stage
- Confirm the expected source document and current version are present in the index.
- Check document identity and metadata used for filtering, access control, and result attribution.
- Inspect the chunk produced from the relevant source location and confirm its text is complete and searchable.
- Verify enrichment and embedding outputs, then confirm the resulting records were persisted and are available to retrieval.
- When an update is missing, find the earliest ingestion stage where the current source diverges from indexed content rather than compensating with broader retrieval.
Microsoft’s RAG pipeline description separates chunking, enrichment, embedding, and persistence, which makes those stages independently inspectable. Freshness and completeness should be verified from the indexed records, not inferred from the source file alone.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Compare the likely fixes with evidence
| Choice | Useful when | Trade-off to measure |
|---|---|---|
| Keyword retrieval | Questions contain exact terminology, names, or identifiers used in the source. | Test whether it misses paraphrases or semantically related wording. |
| Vector retrieval | Questions and source passages express similar meaning with different wording. | Test whether exact terms and identifiers are retrieved reliably. |
| Hybrid retrieval | The workload includes both exact-term and meaning-based matches. | Compare candidate relevance and recall with each constituent mode. |
| Fixed-size chunking | A simple baseline is useful and content units are reasonably uniform. | Check for split context, mixed topics, and overlap-driven duplication. |
| Structure-aware or semantic chunking | Document sections, headings, or coherent semantic units matter to answers. | Measure implementation complexity against passage completeness and retrieval quality. |
| Larger candidate pool, smaller final context | Relevant evidence is missing from a narrow initial search but the model context must remain controlled. | Measure recall against latency, token use, and noise. |
| Reranking | Relevant candidates exist but need more query-aware ordering. | Measure precision gains against added processing latency. |
No option in the table is a universal fix. Use the same versioned questions and expected passages to compare the actual workload.
Keep retrieved content secure and grounded
Retrieved text is input to a model, not a trusted instruction source. A document can contain malicious instructions, so content processing and generation should guard against prompt injection. Apply access controls during retrieval to prevent sensitive indexed information from being returned to unauthorized users. Grounding can reduce unsupported answers, but it does not guarantee correctness; Microsoft Foundry documentation warns that irrelevant or incomplete retrieved passages can still lead to incomplete or inaccurate answers.
Recommended Free Tools
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
