iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
A reported score of 99.95% on LoCoMo, a benchmark for very long conversational memory, sounds like a solved problem. It is not established as one. The figure comes from a company-associated post that says the system’s memory was post-trained into its model weights using the same conversation set the benchmark tests. When a system has learned the exact conversations it is later examined on, the score shows how well it reproduces material it has already absorbed. It does not show how well it remembers conversations it has never seen. LoCoMo remains a useful test, but the headline number cannot carry the weight the title gives it.
What the 99.95% figure does and does not establish
The score appears in a DEV Community repost associated with a company author account. According to that post, the team reported 99.95% on LoCoMo after post-training memory directly into model weights, and it trained on the same conversations the benchmark evaluates. The original post could not be retrieved in full for this review, so its complete methodology, its evaluation configuration, and its publication date cannot be checked. No independent reproduction of that specific result was found.
Read the number as a company-reported result on a non-independent setup. It is not a held-out score, and it is not a neutral measurement of memory performance.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
The catch: the test conversations were part of training
A memory benchmark is only informative if the system being tested has not already seen the material. Its purpose is to ask whether a system can recall details, track events across sessions, and connect distant facts in conversations it encounters for the first time. If the conversations were used to shape the model’s weights, the test partly measures recall of training data. A higher score in that case may reflect the overlap rather than better general memory.
#1 Best Overall
The available material does not quantify how much the overlap inflated the score. No corrected or contamination-adjusted figure can be supported from what is publicly accessible, so the honest position is that the 99.95% result is unverified as evidence of generalization, and its size relative to a clean evaluation is unknown.
What LoCoMo actually measures
LoCoMo was introduced in the ACL 2024 paper “Evaluating Very Long-Term Conversational Memory of LLM Agents” by Adyasha Maharana, Dong-Ho Lee, Sergey Tulyakov, Mohit Bansal, Francesco Barbieri, and Yuwei Fang. The authors built the conversations with a machine-human pipeline. Each conversation is grounded in a persona and an event graph, and human annotators checked long-range consistency and grounding. The paper reports conversations averaging 600 turns and 16K tokens, spanning as many as 32 sessions.
The benchmark covers three task types: question answering, event summarization, and multimodal dialogue generation. Its central finding is a limitation. The paper reports that long-context language models and retrieval-augmented generation can improve results, but they still lag human performance. In the authors’ words:
“Our experimental results indicate that LLMs exhibit challenges in understanding lengthy conversations and comprehending long-range temporal and causal dynamics within dialogues.”
The paper’s benchmark versus the released subset
The scale described in the paper is not the same as the material most readers can download. The official LoCoMo repository from Snap Research reports that its initial March 2024 arXiv release contained 50 conversations. The authors then selected the longest conversations with high-quality annotations, leaving a ten-conversation subset for more cost-effective evaluation of closed-source models. Keep these figures separate when describing dataset size.
| Feature | Original paper (ACL 2024) | Released repository subset |
|---|---|---|
| Conversation count | Not stated in the paper summary reviewed | Ten conversations; the initial March 2024 arXiv release had 50 |
| Conversation length | Averages 600 turns and 16K tokens | Selected as the longest conversations with high-quality annotations; per-subset averages not stated |
| Sessions | Up to 32 sessions | Not stated |
| Tasks with annotations | Question answering, event summarization, multimodal dialogue generation | Question-answering and event-summary annotations; conversations also support multimodal dialogue-generation work |
| Images | Not stated | Images themselves are not released; image URLs, generated captions, and search queries appear in the data |
Why the benchmark still matters
LoCoMo is valuable because it is hard in the ways that matter for long-running assistants. Its conversations are long enough that a system cannot rely on a single prompt window, and the questions often depend on temporal order or cause and effect across sessions. Those are the points where memory systems usually fail, so the benchmark gives developers a shared place to find the failures.
A high score on LoCoMo is therefore a narrow claim. It says a system performed well on this set of questions, under the conditions reported. It does not guarantee that the same system will handle every user, every deployment, or every kind of conversation. The most useful reading of any LoCoMo result is diagnostic: which task failed, which session span was hardest, and whether the setup was clean enough to trust the comparison.
Do these 3 things before closing this tab:
1Clear out junk files and repair common Windows errors2Scan for outdated or missing drivers - takes under a minute3Repair Windows errors before they cause bigger problemsWhat benchmark-style QA leaves unmeasured
Question answering tests whether a system can retrieve a fact when asked for it. Later work argues that this is only one part of conversational memory, and two newer evaluations show where QA-style probing can miss important behavior.
Latent constraints: LoCoMo-Plus
The LoCoMo-Plus paper, published at ACL in 2026, argues that most memory evaluations emphasize surface factual recall. Its framework asks whether a system retains and applies latent constraints such as a user’s state, goals, and values when a later cue does not repeat them directly. A system can answer every factual question correctly and still ignore a constraint the user stated several sessions earlier.
Natural query forms: LoCoMo-Conv
A September 2026 arXiv preprint, LoCoMo-Conv, derives conversational query forms from LoCoMo itself: dialog, implicit, counterfactual, and composed. It reports retrieval and response-quality gaps that standard QA probing does not reveal. The practical point is that a question phrased the way a person actually talks, with indirect references or hypotheticals, can be harder than the same fact asked directly.
Together, these papers mean that success on original LoCoMo question answering does not establish competent memory in live conversation. It is one measurement among several.
Recommended Free Tools
How to check a LoCoMo score before you trust it
When two LoCoMo results are compared, or when a single headline is judged, these six checks separate a meaningful result from a marketing number:
Best Value
- Training and tuning overlap. Confirm whether training, fine-tuning, prompt design, or memory construction used any benchmark conversations. This is the check the 99.95% claim fails on its own description.
- Dataset version and subset. Identify whether the run used the 50-conversation initial release or the ten-conversation released subset, and whether the reported figures refer to the paper’s full scale.
- Task coverage. Establish whether only question answering was run, or whether event summarization and multimodal dialogue generation were included.
- Metric and judge configuration. Look for the scoring metric, the judge model if one was used, and the version of that judge.
- Retrieval, generation, and end-to-end quality. Check whether the claim covers retrieval accuracy, generated answers, or the full response in conversation, since these can diverge.
- Reproducibility. Prefer results that can be rerun from a public evaluation harness. A score that cannot be reproduced should be labeled as reported, not verified.
The 99.95% claim, as described, fails the first check and cannot be assessed on the others from the accessible material. Until those details are published, the score belongs in the category of company claims rather than benchmark results.
LoCoMo still gives the field a common, demanding test of long-range conversational memory. Its value depends on keeping training and evaluation separate, and on reporting which parts of the benchmark a result actually covers.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Clear out junk files and repair common Windows errorsFree Scan →
