Free tools Windows power users keep installed
One-click scans. No signup required.
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
In a vendor-run test of 528 questions, Jylus reports that its Context Pack used an average of 2,532 input tokens and achieved 100% observed strict accuracy, compared with 205,129 tokens and 78.79% accuracy for full context. That is 98.77% fewer input tokens than the full-context baseline. The result is striking, but it describes one frozen benchmark—not proof that shrinking an LLM’s prompt will generally improve accuracy.
What the 98.77% reduction compares
Josh Hodgetts, founder of Jylus, published the comparison on DEV Community on October 5, 2026. The company says all three approaches used the same Gemini 3.1 Flash Lite settings, questions, and deterministic scorer. Its reported averages were:
| Evidence path | Average input tokens | Strict accuracy |
|---|---|---|
| Full context | 205,129 | 78.79% |
| BM25-only RAG | 8,128 | 76.14% |
| Jylus Context Pack | 2,532 | 100% observed (528/528) |
The 98.77% figure is the reduction from the full-context average to the Context Pack average. It is a comparison of average model input tokens, not a claim that every question used exactly those amounts. The figures and scoring results are vendor-reported; the benchmark has not been independently reproduced. Hodgetts’s benchmark article describes the results and its limits.
Why fewer tokens do not explain the accuracy increase
The comparison changes more than prompt length. Full context sends a broad set of material to the model; BM25-only retrieval selects material through keyword matching; Jylus says its system also prepares and organizes evidence before sending a smaller context. The vendor describes retrieval, state and relationship resolution, and a compiled Context Pack with source references that can retain conflicts or gaps. The model then reasons over that prepared evidence.
#1 Best Overall
Because those evidence-preparation steps differ, the three-arm result cannot identify which change produced the higher score. Token reduction alone is not enough to account for the outcome: BM25-only RAG reduced average input from 205,129 to 8,128 tokens but scored 76.14%, below the full-context result of 78.79%. That is a result for this BM25 setup, not a verdict on every retrieval-augmented generation system.
What “100% observed” means
Jylus reports strict-scoring passes on all 528 questions in its frozen benchmark. “100% observed” therefore means 528 out of 528 for that test; it is not a universal accuracy guarantee or evidence that the system will answer every real-world question correctly. The workload, scoring rules, and model settings define what the number says.
The vendor’s benchmark methodology page describes 528 unique adversarial questions across four data domains and 11 adversarial classes, with 1,584 completed evaluations. It says the same model settings and scorer were used across the three paths and that raw answers, provider request IDs, timings, hashes, and checkpoints were retained. The page also says ranking, fusion, evidence selection, temporal and relationship processing, optimization, and tuning are proprietary. Retained evaluation records may help document a test, but they do not amount to independent reproduction of its result.
Why temporal evidence is a difficult test
Many factual questions depend on when a record was true, when it was entered, or whether a later record superseded it. Imagine a subscription record with an old price, a replacement price, and a correction entered later that applies to an earlier billing event. “What is the price now?” and “What price was known at the time?” are different questions. A system that retrieves only the most similar text can miss that distinction.
For a reliable answer, the evidence needs to make several things clear: which source supports the claim, what period the fact applies to, whether another record supersedes it, whether sources disagree, and whether the available evidence is sufficient to answer. Hodgetts uses examples involving support histories, changing subscriptions, and incident investigations. These are sensible use cases for testing temporal scope and provenance, but the published benchmark does not establish performance across all such applications.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How to judge the result for an AI application
The benchmark is useful as a prompt to evaluate evidence preparation, not as a guarantee to import into a product decision. A practical comparison should hold questions, model settings, and scoring rules constant while inspecting what evidence each approach provides.
- Check answer quality: Define strict scoring before testing, and review the failures as well as the aggregate score.
- Measure the input: Record average tokens for the complete workload and note how token counts vary across question types.
- Audit evidence coverage: Verify that selected records preserve dates, provenance, superseding updates, disagreements, and explicit gaps.
- Test temporal questions: Include both “what is true now?” and “what was known or true then?” cases, including corrections entered after the event.
- Assess reproducibility: Determine whether the evidence-selection logic, evaluation set, and scoring process can be inspected or independently tested.
- Retest on your own data: A result on one vendor’s frozen workload does not establish expected performance on a different domain or production traffic.
Hodgetts summarizes the attribution limit this way: “That does not isolate which part of evidence preparation caused the improvement, but it gives us something concrete to investigate.” The appropriate conclusion is similarly bounded: this benchmark reports a large token reduction alongside higher observed accuracy, while leaving the cause and generalizability unproven.
Recommended Free Tools
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
