Reproducing results across AI agent memory systems is hard, different LLMs, embeddings, token budgets, and scoring methods make comparisons almost meaningless.
We built MemEval, an open-source benchmark that evaluates memory systems under standardized conditions and tracks token efficiency. While benchmarking, we discovered recurring failure modes, which led to PropMem, a factual memory system designed to address them efficiently.
Both projects are Open Source: ready for evaluation, extension, or collaboration.
I built AMB, an open and preregistered benchmark for memory layers in coding agents.
Most memory benchmarks test whether a system retrieves a relevant chunk. AMB tests whether that memory actually helps an agent complete real coding work, or causes a worse change.
The agent runs Claude Code inside a sandboxed repository. Executable tests grade the resulting artifact. The benchmark is adversarially constructed with absent, stale, superseded, contradictory, adjacent, and irrelevant memories.
The current public feed contains 195 pre-authored session transcripts. A separate hard corpus record scales to 4,900 documents, including 196 real documents and 4,704 synthetic distractors, with roughly 143,000 chunks.
I am inviting small memory vendors to test their systems. There is no participation fee. Vendors can review the adapter contract, run with their own credentials, and publish their configuration and results. Negative results are welcome.
The current suite mainly measures the read path. It does not yet measure whether a memory layer learns from the agent’s own work across sessions, and no multi product ranking has been published.
Repository:
I would especially like feedback from maintainers of memory layers, MCP servers, agent frameworks, and retrieval systems.
What would you want this benchmark to measure before trusting its results?
I really like the move from “did the system retrieve something related?” to “did that memory help the agent make a correct change?” That is where memory benchmarks stop being a fancy scavenger hunt and start becoming useful.
The stale, superseded, contradictory, and irrelevant cases are especially interesting to me. One dimension I would consider adding is whether the memory was still eligible to influence the action.
A retrieved memory can be relevant and still have the authority of a visitor badge that expired yesterday. Relevance is not a hall pass.
I have been working with small synthetic cases that make that distinction explicit: a memory can be current, historical, held, denied, or unresolved based on time, scope, provenance, conflict, and supersession. The result includes a receipt so the question is not only “what did the agent do?” but also “why was this state allowed to shape the action?”
I would be curious whether AMB could add an “eligible-state use” dimension alongside retrieval and task outcome:
correct use of eligible memory;
correct withholding of ineligible memory;
action influenced by stale, superseded, or unsupported memory; and
over-withholding when the memory was still valid.
That seems like a useful bridge between memory quality and governance: the agent may produce working code, but was it allowed to trust the state that got it there?