The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREEClear out junk files and repair common Windows errorsFree Scan →iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
On one 94-page commercial property insurance policy, LlamaIndex’s SentenceSplitter had the highest reported retrieval recall@5, while Chonkie’s RecursiveChunker split the document fastest. LangChain’s token-aware recursive splitter scored close to the leader. These are results from one author’s small, task-specific experiment—not a universal ranking of the libraries.
What the 94-page PDF test found
The comparison used a commercial property insurance policy converted with pdftotext -layout into 214,883 characters. The author wrote 38 questions by hand, each paired with an answer string said to occur on exactly one page. The sample question was: “what’s the deductible for wind damage during a named storm.”
For retrieval, the author embedded text with BAAI/bge-small-en-v1.5, normalized the vectors, ranked them by cosine similarity, and checked the top five chunks. Recall@5 is the share of questions for which at least one of those five chunks contained the paired answer string. The author reports spending 47 minutes labeling the questions.
| Library and configuration | Recall@5 on 38 questions | Time to chunk the PDF | Chunks produced |
|---|---|---|---|
LangChain: RecursiveCharacterTextSplitter.from_tiktoken_encoder, size 512, overlap 64 |
0.79 | 2.8 seconds | 137 |
LlamaIndex: SentenceSplitter, size 512 tokens, overlap 64 |
0.86 | 4.1 seconds | 129 |
Chonkie: RecursiveChunker, size 512 |
0.82 | 0.38 seconds | 133 |
All figures in the table are reported by the comparison’s author, not independently verified measurements. The test measures whether a retrieved chunk contains a known answer string; it does not measure generated-answer quality, faithfulness, or complete application performance. Read the author’s comparison and scoring details.
#1 Best Overall
Why chunk-size units changed the comparison
The author’s first LangChain trial used 512 characters, while the other libraries were configured for 512 tokens. That LangChain run scored 0.44 recall@5. After changing it to from_tiktoken_encoder(chunk_size=512, chunk_overlap=64), it scored 0.79. The corrected result is the one in the table; the initial result is not an apples-to-apples comparison because character and token limits are different units.
Even when two settings both say “512 tokens,” they may not count tokens identically. The LangChain configuration uses tiktoken’s default cl100k_base, which is not the tokenizer used by the BERT-family embedding model in this experiment. Token-count drift can matter more for code, JSON, and non-Latin text. Record both the size unit and tokenizer when evaluating your own corpus.
Rank #2
Chunk size and overlap changed recall
On the same labeled policy test, the author’s chunk-size sweep reported these recall@5 results:
| Chunk size | Reported recall@5 |
|---|---|
| 256 tokens | 0.74 |
| 512 tokens | 0.86 |
| 768 tokens | 0.81 |
| 1,024 tokens | 0.68 |
The overlap sweep reported 0.79 recall@5 at zero overlap, 0.86 at 64 tokens, and 0.87 at 128 tokens. The 128-token setting generated 22% more chunks to store and search. These are results for this small test, not recommended defaults: the right size and overlap depend on the document structure, retrieval questions, and cost of indexing and searching additional chunks.
What the results do—and do not—say about each tool
LlamaIndex: strongest recall in this test
Its SentenceSplitter produced the highest reported recall@5, 0.86, with 129 chunks. It took 4.1 seconds to chunk the document, the slowest of the three in this run. That makes it the retrieval leader for this particular labeled policy evaluation, not proof that it will lead on other document types or workloads.
Chonkie: fastest single-document split
Its RecursiveChunker took a reported 0.38 seconds and produced 133 chunks, with recall@5 of 0.82. This is the fastest chunking result in the single-PDF comparison. It says nothing by itself about total pipeline speed, which can include parsing, embedding, indexing, and other work.
LangChain: close retrieval result with a token-aware configuration
With the corrected token-aware configuration, LangChain scored 0.79 recall@5 and produced 137 chunks in 2.8 seconds. Its score was below the other two reported scores in this test, but the gap is specific to these questions and this policy. The initial character-based run should not be used to characterize the corrected configuration.
A separate corpus run is not the same benchmark
The comparison’s author also reports a run over 1,240 files containing 2.1 GB of extracted text: Chonkie took 47 seconds, LangChain 3 minutes 4 seconds, and LlamaIndex 6 minutes 12 seconds. The account does not document enough detail to equate this run with the 94-page test or generalize the times to another machine, corpus, or version. Treat it as a separate author-reported result, not a controlled extension of the PDF comparison.
Best Value
Choose by your workload, not a one-test ranking
Chunking is one part of a retrieval-augmented generation system. Your decision should reflect what the application needs and should be based on measurements from the documents and questions it will actually handle.
- For retrieval quality: create representative questions with known answers and measure recall@k or another retrieval metric suited to the task. A labeled evaluation shows whether useful evidence reaches the model; it does not certify the quality of the model’s final answers.
- For ingestion throughput: time chunking at your expected corpus size on the hardware you plan to use. Keep parsing and embedding timings separate so a fast splitter is not mistaken for a fast end-to-end pipeline.
- For chunk semantics: write down whether size means characters or tokens, the tokenizer used, and how overlap is counted. Check alignment with the model and content you use.
- For architecture fit: LangChain’s own comparison presents LangChain and LangGraph as oriented toward agent orchestration, while it describes LlamaIndex as oriented toward document parsing, indexing, and retrieval. It suggests combining them when both roles are needed. This is the vendor’s framing, not a finding of the PDF benchmark. See LangChain’s comparison.
- For Chonkie context: its repository describes it as a lightweight ingestion library for RAG pipelines. Its separate project benchmarks use other datasets and hardware, and the project cautions that results vary by setup; they do not independently validate the 94-page PDF results. Chonkie repository and Chonkie benchmark documentation.
How much confidence to put in the reported numbers
The 94-page experiment is one author’s report on one policy and 38 hand-labeled questions. The article publishes a scoring script and splitter configurations, but the PDF and the question-and-answer-needle dataset are not available in the reviewed account. That means the exact numerical results cannot be independently reproduced from those materials.
Use the comparison as a useful example of how retrieval outcomes and splitting speed can diverge, then run a labeled evaluation on your own content. Library APIs, defaults, package sizes, and tokenizer behavior can change, so check the versions and settings you intend to deploy rather than assuming the reported setup still matches them.
Recommended Free Tools
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
