iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
For the setup Haoning Kan tested, multi-target ListWise + Rewrite offered the best balance: it could update several outdated memories at once while retaining useful details. The highest benchmark score alone did not establish the best user experience. The comparison shows why memory systems need to be judged on how they find and revise stale facts, not just on whether a benchmark answer comes out right.
What the six configurations changed
These were not six different memory architectures. They were combinations of two choices: how a new extracted fact is compared with existing memories, and what happens when a match is found.
- Judgment method: PointWise checks candidate memories pair by pair; ListWise considers a batch together. Add-Only skips the deduplication judgment.
- Update handling: Replace overwrites a matched memory; Rewrite merges the new fact with relevant prior detail.
- Threshold: The threshold controls which candidates are considered similar enough to send for judgment. A judgment model cannot resolve a contradiction it never sees.
Kan describes mem0 v2.0.7 as having no standalone deduplication in the open-source path he examined, MemOS 2.0.23 as using pair-by-pair PointWise judgment, and signetai 0.123.22 as using batch ListWise judgment with replacement. Those are product examples, not the products’ direct scores in the table below: Kan reproduced the strategy comparisons on NeatMem to reduce differences from product-specific prompts and retrieval implementations.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
How the six strategies scored and how many calls they used
The figures below are means across five LoCoMo runs reported in Haoning Kan’s DEV Community article, displayed as “Sep 26”; examples and comments in the article point to 2026, though the page does not explicitly state a year. These are author-reported results, not an independent replication. Every configuration used 1,400 extraction calls.
#1 Best Overall
| Configuration | Dedup calls (mean) | Merge calls (mean) | Total calls (mean) | LoCoMo score |
|---|---|---|---|---|
| Add-Only | 0 | 0 | 1,400 | 90.68% |
| PointWise + Rewrite, threshold 0.8 | 2,018 | 576 | 3,994 | 90.27% |
| PointWise + Rewrite, threshold 0.4 | 23,985 | 1,801 | 27,186 | 89.13% |
| ListWise + Replace, threshold 0.4 | 5,021 | 0 | 6,421 | 89.52% |
| ListWise + Rewrite, threshold 0.4 | 4,997 | 366 | 6,763 | 90.47% |
| ListWise + Rewrite, threshold 0.8 | 1,263 | 183 | 2,846 | 90.75% |
The test used the same write, embedding, and scoring models in each configuration, started each run with an empty store, retrieved the top 200 memories, and disabled reranking. Prompts followed mem0’s evaluation setup. Kan used NeatMem v0.5.8 with BM25 on, entity extraction off, and thinking off. Thresholds in this experiment were uniform NeatMem cosine thresholds; values from different vendors are not directly comparable.
Why the best score does not automatically mean the best experience
Kan reports that the scores fell within a narrow band and says their differences were comparable to run-to-run variance. In this setup, retrieving up to 200 memories could leave duplicate or outdated entries alongside the correct one; the answer model might still find enough information to answer correctly. Benchmark accuracy can therefore hide a poorly maintained memory store.
Rank #2
A threshold can hide the contradiction from the judge
In Kan’s Seattle-to-Austin example, the new location contradicted the old one, but the similarity between the candidate memories was reported at roughly 0.71–0.75. With a candidate threshold of 0.8, PointWise did not send the pair to the language-model judge. Both facts remained in memory. Lowering the threshold can expose more potential matches, but it also increases judgment work: in this comparison, PointWise + Rewrite at 0.4 made 27,186 total calls on average, versus 3,994 at 0.8.
Recommended Free Tools
Replacement can discard details that are still useful
ListWise + Replace reduced judgment calls by checking candidates in a batch, but overwriting the old entry could remove relevant context. In the breakfast example, replacing a coffee routine dropped an older detail about listening to Spanish podcasts. Rewrite combined the changed routine with the earlier information instead of discarding that detail.
One judgment may need to update more than one old memory
In the tea example, single-target ListWise + Rewrite updated one old item even though the new fact affected several; an old coffee-shop memory was left in the present tense. Multi-target ListWise + Rewrite considered multiple candidates in one judgment and used one merge call to combine the update with both stale items.
What Kan recommends—and what the result does not prove
For the protocol he tested, Kan recommends multi-target ListWise + Rewrite. Batch judgment keeps the number of judgment calls relatively low, handling multiple affected entries avoids leaving stale facts behind, and rewriting preserves useful details better than wholesale replacement. His recommendation is specific to this comparison; the results do not establish that the same configuration is best across products, models, retrieval systems, or tasks.
Rank #4
Kan’s conclusion captures the distinction: “A higher score doesn’t mean a better experience.” A benchmark can reward getting an answer right without revealing whether the store contains conflicting facts, whether updates preserve context, or how costly the update process is.
Why memory benchmarks can disagree
Different benchmarks test different kinds of memory use. EvoMemBench evaluates four combinations: memory within an episode or across episodes, and memory containing knowledge or execution experience. Its authors report that memory can help when context is insufficient, tasks are difficult, or stored experience matches the target decision process. It can also hurt when retrieved evidence is irrelevant, compression removes execution details, or a procedure is transferred to a mismatched task. No one memory form works consistently across all settings.
The Memory Papers benchmark overview likewise cautions against treating a score as a universal measure of memory quality. It distinguishes, for example, LoCoMo’s conversations of up to 32 sessions, LongMemEval’s multi-session question answering with 500 questions, and MemoryArena’s interdependent agent tasks. Scores depend on the task, model, answer and evaluation setup, memory horizon, and retrieval configuration.
A separate vendor report illustrates why scores across benchmarks should not be treated as a direct leaderboard. Redis’s September 21, 2026 report says its Remis + Instruct strategy achieved 86.5% task-averaged accuracy on LongMemEval, while its Vertex Memory Bank test scored 42.4%. The report describes a 500-question benchmark with about 24 prior sessions per question on average and a GPT-4o binary judge. Those vendor-reported results use a different dataset and setup from Kan’s NeatMem comparison, so they do not rank the six configurations against Redis’s systems.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.How to apply the findings to an agent memory system
Use the comparison as a way to diagnose the failure mode you care about, rather than as a universal recipe. In particular:
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →- If contradictions survive updates, check candidate retrieval and thresholding as well as the judgment prompt. A better judge cannot act on a missing candidate.
- If updates erase useful context, compare merge behavior with replacement using examples where an old memory contains both changed and still-valid details.
- If multiple memories can be invalidated by one fact, test whether the update path can identify and revise more than one target.
- If a benchmark score changes only slightly, consider repeated runs and inspect stored memories and call volume before treating the difference as meaningful.
- If the agent performs procedures, evaluate whether stored experience matches the decision process at hand; a knowledge-oriented retrieval result may not predict execution quality.
The practical lesson is to evaluate memory as a lifecycle: extraction, candidate selection, judgment, update, retrieval, and task outcome. A high answer score is useful evidence for that tested task, but it is not by itself proof that the agent’s long-term store is accurate, economical, or pleasant to use.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
