What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
In one small codebase retrieval pilot, a reader model answered more questions correctly when given a whole retrieved document than when given the top 10 retrieved chunks. The result varied by query type: whole documents did much better on code questions, while chunks did slightly better on prose questions. The experiment is useful as a warning against assuming that smaller chunks are always better context, not as proof that whole-document retrieval wins generally.
What did the experiment test?
Mikhail’s September 2026 F5 experiment examined whether the unit of retrieved context changes how well a reader model answers questions about a real codebase. It used 16 frozen queries—eight about code and eight about prose—and four evaluation arms. Each query-arm combination was tried 10 times, producing 640 judged verdicts. The author reports that each evaluation used a reader call and a separate judge call, with pinned model IDs; the findings therefore describe this particular setup, not every retrieval system or model. Mikhail’s experiment and results.
| Arm | Context supplied to the reader | What it helps assess |
|---|---|---|
| A | The top 10 retrieved chunks | Answering from chunk-based retrieval |
| B | The full top-1 retrieved document | Answering from document-based retrieval |
| C | The gold file, supplied as oracle context | Answering when the known relevant file is provided |
| D | Empty context | Answering without retrieved context |
The oracle arm is a diagnostic comparison, not a retrieval result: it starts with the gold file already selected. The empty-context arm likewise tests a different condition from either retrieval strategy.
What were the overall answer results?
Across the 160 trials in each arm (16 queries × 10 trials), the whole-document arm had the highest judged answer rate among the two retrieval arms. The figures below are author-reported results from this pilot, not independent measurements. Source: Mikhail (2026).
#1 Best Overall
| Arm | Correct verdicts | Rate | Queries with a strict majority of correct trials |
|---|---|---|---|
| Top-10 chunks | 26/160 | 16.3% | 2/16 |
| Full top-1 document | 55/160 | 34.4% | 5/16 |
| Gold-file oracle context | 156/160 | 97.5% | 16/16 |
| Empty context | 0/160 | 0% | 0/16 |
A strict majority means more than half of a query’s 10 trials were correct. A 5–5 split is a tie, not a majority. That distinction matters: the whole-document arm’s 55 correct verdicts overall did not mean it consistently won across most individual questions.
Did the result hold for both code and prose?
No. The split by query type was sharply different in this 16-query set. Mikhail reports the following answer rates for the two retrieval arms; each subset contains 80 trials (eight queries × 10 trials). Source: Mikhail (2026).
| Query subset | Top-10 chunks | Full top-1 document |
|---|---|---|
| Code | 5/80 (6.3%) | 40/80 (50.0%) |
| Prose | 21/80 (26.3%) | 15/80 (18.8%) |
For code questions, whole-document context had a much higher answer rate in this run. For prose questions, chunks had a modestly higher rate. The author reports Wilson 95% intervals for these comparisons: the code-arm intervals did not overlap, while the prose-arm intervals did. With only eight queries per subset, that interval pattern should not be read as independent replication or a general guarantee; the prose result is direction-only.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →What did retrieval itself find?
Retrieval metrics describe whether a relevant item was found; judged answer rates describe whether the reader answered correctly after receiving context. They are different stages and should not be conflated.
- For chunk retrieval, hit@1 was 5/16, and hit@3 and hit@10 were each 6/16. The author says no gold item appeared beyond the top three in this run.
- For document retrieval, the top-1-is-gold rate was 5/16.
- For the code subset, retrieval hit@1 was 4/8; for prose, it was 1/8.
The equal 5/16 counts for chunk hit@1 and document top-1-is-gold do not mean the same retrieval events occurred. Nor do those counts alone explain the difference in answer rates: retrieval quality and what the reader can do with the returned context both matter. These retrieval figures are reported by Mikhail (2026). Experiment details.
Does this show that context completeness matters more than context size?
It suggests that possibility, but does not isolate it. The author reports that oracle context averaged fewer characters than the full-document context while scoring much higher. That is consistent with the idea that relevant context can matter more than sheer volume.
However, the comparison between top-10 chunks and full documents changes more than one thing at once: the returned unit and its size are confounded. The pilot did not include a whole-document-versus-gold-span-sized-excerpt control. Without that comparison, it cannot establish whether the advantage came from including more of a document, from the way the information was packaged, or from another difference in the setup.
What are the experiment’s limits?
- Small query set: the results come from 16 frozen queries, including only eight prose questions. The author characterizes the percentages as directional rather than absolute effects.
- Snapshot, not hard-frozen index: the index was snapshotted, and drift between runs was documented but not eliminated.
- No token-cost attribution for F5: context character counts were recorded, but the author did not report token costs for the F5 arms.
- No reported judged-noise measurement or frontier-reader rerun: the article does not establish how much results might change under those checks.
- Unresolved unit-versus-volume control: a gold-span-sized excerpt comparison is still needed to separate the effect of returned unit from the amount of context.
These limits keep the takeaway narrow: in this harness and query set, full-document context outscored top-10 chunks overall, especially on code questions. The pilot does not establish a universal retrieval rule. Mikhail’s account includes the reported setup and caveats.
Best Value
What does the separate NodeRAG side experiment add?
It is a distinct, no-LLM experiment over 10 rule queries, not another arm of F5. Mikhail reports chunked TF-IDF finding 8/10 rule queries and graph BFS finding 7/10. The graph approach had a lower estimated token total, 170,140 versus 301,981, but missed three queries; this run therefore does not show graph traversal winning.
The token totals are estimates from the harness, calculated using word count multiplied by 1.3—not measured model-token usage. They should not be compared with F5 token costs, which were not reported. Source: Mikhail (2026).
How should a codebase retrieval team use this result?
Treat the pilot as a reason to test the returned unit against your own questions, not to switch every system to whole files. A useful evaluation should keep retrieval and answer quality separate: record whether the relevant file or passage was retrieved, then judge whether the reader produced a correct answer from the supplied context. Break results out by code and prose questions so an aggregate rate does not conceal different behavior.
Quick wins for a faster PC:
Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →For the specific question, “where is the real business logic here, and what can I safely throw away?”, this experiment supports caution about discarding context before testing what the reader needs. It does not tell teams how to choose chunk size, document boundaries, or retrieval policy for a different codebase. Anthropic’s Contextual Retrieval and Jina’s Late Chunking are mentioned in the source as related approaches, but neither was an arm in this F5 experiment. Source: Mikhail (2026).
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
