82.0% vs 56.4% with the same reader, official judge, and ingestion.
MemBukkit benchmark lab · open reproduction 92.6% on LongMemEval-S.
92.6% on LongMemEval-S.
When LongMemEval judges, MemBukkit leads.
Memseek is our context engine. MemBukkit is our open agent-memory showcase and benchmark harness. The SOTA-performing search method demonstrated there will be available as a search method in the hosted version of Memseek.
LongMemEval-S · 500 questions · frozen recipe · official benchmark judge
frozen, named recipes
Agent-memory results you can rerun.
These are the claims registered in MemBukkit. Each recipe pins the reader, distiller, judge, encoder, routing flags, and output directory. Different protocols are labeled; they should not be blended into one ranking.
| Benchmark | Frozen recipe | Reader / distiller | Scoring protocol | Expected |
|---|---|---|---|---|
| LongMemEval-S | longmemeval-gpt54 | gpt-5.4 reader + distiller | official gpt-4o judge | 92.6% |
| LongMemEval-S | longmemeval-gemma | Gemma 4 26B reader + distiller | official gpt-4o judge | 88.8% |
| LongMemEval-S | longmemeval-gpt4o-mini | gpt-4o-mini + fine-tuned encoder | official gpt-4o judge | 82.0% |
| LoCoMo | locomo-mem0 | gpt-4o-mini + fine-tuned encoder | Mem0 reader + judge protocol | 87.5% |
LongMemEval and LoCoMo report answer accuracy. Reproductions pass within ±3 percentage points because LLM readers and judges are nondeterministic. “Official judge” applies to the three LongMemEval rows; LoCoMo deliberately follows the separate Mem0 protocol.
LongMemEval-S · who judges what
Same benchmark. Different judges. MemBukkit leads under the official one.
Higher reported scores exist, but they do not all use LongMemEval’s official gpt-4o judge. The pattern and protocol columns keep that distinction visible instead of flattening every number into one leaderboard.
Open the current MemBukkit comparison ↗| System | Reported accuracy | Judge protocol |
|---|---|---|
| OMEGA + GPT-4.1 | alternate judge GPT-4.1 answers + grades | |
| Mem0 Cloud + GPT-5 | alternate judge author's GPT-5 judge | |
| MemBukkit + gpt-5.4 | official judge official gpt-4o judge | |
| Hindsight | alternate judge GPT-OSS-120B judge | |
| Mem0 OSS | alternate judge author's GPT-5 judge | |
| Supermemory | official judge official gpt-4o judge | |
| Zep | official judge official gpt-4o judge | |
| Full-context reading | official judge official gpt-4o judge |
Restricted to systems scored by the official gpt-4o judge, MemBukkit’s 92.6% is the highest published result in the current field. Patterned rows use another judge and are included as context, not as like-for-like rankings.
~3.2k tokens read per question instead of roughly 100k.
Accuracy falls from 80.0%; excluding a matched random set leaves 82.3%.
long-horizon scale
BEAM, from 100K to 10M tokens.
The Gemma recipes use BEAM’s vendored official judge and report its average across nine ability categories. That metric is not LongMemEval accuracy.
100K 0.535
beam-100k-gemma 1M 0.498
beam-1m-gemma 10M 0.447
beam-10m-gemma the search method
Search is the shared capability—not the product name.
MemBukkit exposes the method in a small, inspectable agent-memory system. Memseek makes that search available inside a broader context engine that also maintains state, provenance, policies, and prompt-time context.
Read the implementation walkthrough →- 01PartitionOrganize evidence into topic buckets.
- 02RouteOpen the best buckets under a scan budget.
- 03RankFuse semantic and relevance signals.
- 04AnswerRead the selected evidence with receipts.
separate multi-hop search evaluation
MuSiQue · 2WikiMultiHopQA · HotpotQA
MemBukkit also ships the research evaluator for the SOTA search path: shared public splits, HippoRAG-compatible EM/F1 and Recall@k scoring, and zero LLM calls at index time. This evaluator is separate from the frozen agent-memory recipes above.
$ membukkit rag-eval \
--datasets musique,2wiki,hotpot \
--methods dense,coremem \
--embedder nvidia/NV-Embed-v2 \
--coremem-encoder nvidia/NV-Embed-v2 \
--coremem-decompose reproduce it
Start cheap. Then run the full recipe.
The lite run validates keys, dataset download, and model weights on a small subset. It
is a smoke test, not a comparable score. Drop --lite for the published run,
then use --check against the frozen tolerance band.
terminal
# install the open showcase
$ git clone https://github.com/memseekai/membukkit
$ cd membukkit
$ pip install -e ".[all]"
$ export OPENAI_API_KEY=sk-...
# inspect every frozen recipe
$ membukkit bench --list
# low-cost smoke test
$ membukkit bench --repro longmemeval-gpt4o-mini --lite
# published 92.6% recipe, then verify
$ membukkit bench --repro longmemeval-gpt54
$ membukkit bench --repro longmemeval-gpt54 --check pick the right door
Inspect the memory system. Build with the context engine.
Use MemBukkit to explore the method, run demos, and reproduce the evidence. Use Memseek when you need that search inside a governed context layer for your agents.