Why benchmark AI memory at all?
Because "it remembers" is easy to demo and hard to prove. A memory layer can look great on three hand-picked questions and quietly miss the fourth. Public benchmarks put hundreds of questions with known answers in front of it, so the miss rate shows.
Two benchmarks have become the standard for long-term memory: LongMemEval (published at ICLR 2025) and LoCoMo (published at ACL 2024). Almost every memory product quotes one or both.
What is LongMemEval?
500 questions about long chat histories between a user and an assistant, each with the answer hidden somewhere in dozens or hundreds of past sessions.
| Variant | History per question | Fits in a context window? |
|---|---|---|
| LongMemEval-S | About 50 sessions, roughly 500 turns | In large models, yes |
| LongMemEval-M | About 500 sessions, around 1.27 million tokens | No current model |
The questions cover six kinds of memory work:
Facts the user stated
"What's my dog's name?" from one session.
Facts the assistant said
Recalling the assistant's own earlier answer.
Preferences
Using what the user likes when answering.
Knowledge updates
The fact changed later; use the newest version.
Temporal reasoning
"How long after X did Y happen?"
Multi-session
Combining evidence spread across several chats.
Thirty of the 500 are abstention questions: the answer isn't in the history, and the right response is to say so. They catch systems that guess confidently.
What is the LoCoMo benchmark?
10 very long conversations between two people, with 1,986 questions about them.
| Category | Questions | What it tests |
|---|---|---|
| Single-hop | 841 | One fact from one turn |
| Temporal | 321 | When something happened |
| Multi-hop | 282 | Combining facts across sessions |
| Open-domain | 96 | Common-sense inference about the people |
| Adversarial | 446 | Questions with no valid answer |
Most published results score the first four categories, 1,540 questions, and leave the adversarial category out. One trap: the dataset labels categories with numbers that don't follow the order papers list them in, so it's easy to report "multi-hop" with the wrong questions. Check the mapping against the data itself.
Each LoCoMo conversation is about 20,000 tokens, so it fits in any modern model's context. That makes "give the model the whole conversation" a fair and important baseline.
Retrieval recall vs answer accuracy: what's the difference?
Recall measures the search. Accuracy measures the answer.
| Retrieval recall (e.g. recall@5) | Answer accuracy (QA) | |
|---|---|---|
| Question asked | Was the evidence in the top results? | Was the final answer correct? |
| Needs an LLM? | No | Yes: one to answer, one to judge |
| Typical numbers | Often high 90s | Lower, and depends on the models used |
Both are useful. Recall tells you the memory found the right material; accuracy tells you the whole system worked. Just never compare one product's recall with another's accuracy.
How do LLM judges work, and can you trust them?
A second model compares each answer with the gold answer and labels it right or wrong.
LLM judges are lenient on wording (that's the point: "May 7th" and "7 May" are both right), but they're still models with their own biases. Two simple checks make a judged score much more credible: grade the same answers with a judge from a different model family and report how often they agree, and publish every answer and verdict so anyone can re-grade them.
8 questions to ask before you trust a memory benchmark claim
Ask these of every number, ours included.
- Recall or accuracy? Which kind of score is it?
- Which variant and categories? LongMemEval-S or M? LoCoMo with or without adversarial questions?
- Which models answered and judged? A stronger reader raises the score without changing the memory.
- Compared with what? Is there a full-context baseline on the same model, or only other products' self-reported numbers?
- At what cost? Tokens per question and price, not just accuracy.
- Is the judge checked? A second judge from another model family, with agreement reported.
- Was it tuned on the test? Settings chosen on a held-out half are more honest than settings tuned on the whole set.
- Can you verify it? Published harness, per-question answers and verdicts.
How does our memory score?
Here's ours, held to the same eight questions.
| Benchmark | Score | Setup |
|---|---|---|
| LongMemEval-S, accuracy | 94.2% | Claude Sonnet reads and judges; 93.4% with a GPT judge, 98.8% agreement |
| LongMemEval-S, recall@5 | 98.3% | Session with the answer in the top five, 470 answerable questions |
| LongMemEval-M, accuracy | 83.0% | ~1.27M tokens per question; reader sees about 98% less text |
| LoCoMo, accuracy | 86.6 | 1,540 questions; same model reading the whole conversation scores 89.3 |
| LoCoMo, cost | $0.73 | vs $3.36 for the whole conversation, same model |
Full context beats memory on LoCoMo when the conversation fits, and we say so: memory reaches 97% of that score with 18% of the tokens. On LongMemEval-M the history doesn't fit at all, and memory is the only option. The harness, every answer and every verdict are in our repository. More on the trade-off in long context vs RAG vs memory.
Frequently asked questions
What is LongMemEval?
A benchmark of 500 questions about long user–assistant chat histories, in an S size (~50 sessions per question) and an M size (~500 sessions, ~1.27M tokens).
What is the LoCoMo benchmark?
10 very long two-person conversations with 1,986 questions in five categories; most results score four categories, 1,540 questions, with an LLM judge.
What's the difference between recall and answer accuracy?
Recall: was the evidence found? Accuracy: was the final answer right? Don't compare one with the other.
Are scores comparable between products?
Only with the same answering model, judge, question set and categories. Otherwise read them as direction, not a precise ranking.
What is a good LongMemEval score?
Strong memory systems published in 2026 cluster in the low-to-mid 90s on LongMemEval-S. Ours is 94.2% (93.4% with a GPT judge).
Our figures: LongMemEval (cleaned release) and LoCoMo, measured October 2026 with the same model answering and judging within each comparison. Dataset sizes and categories as published by the benchmarks' authors.
Benchmarked in public. Every answer published.
Open Agentic Brain and Memory is an open-source, self-hosted memory for AI coding agents. The benchmark scripts, per-question answers and judge prompts ship in the repo, so you can rerun every number on this page.
curl -fsSL https://raw.githubusercontent.com/rohithhn/Open-Agentic-Brain-and-Memory/main/install.sh | bash