Agent Memory Benchmark

mnemo measures what a memory system can answer after ingesting email and chat, and what ingesting and answering cost. It reports quality and cost separately across two synthetic corpora.

Table of Contents

• What A Run Looks Like
• What Is In The Corpora
• What It Asks
• Getting Started
• Reading A Result
• Published Results
• Troubleshooting
• Cleanup
• For Contributors
• Project Structure
• Configuration
• Adding Your System
• How The Runner Treats Your Adapter
• Verification
• Provenance And License

What A Run Looks Like

The offline oracle scores 1.0 on a six-document fixture, then the suite passes. Searchable form of the result: accuracy overall: 1.0, every question type at 1.0, and deferred to judge: 0. Note that freshness splits into two lines — a system must not only be current, it must resist a stale value still sitting in the corpus.

What Is In The Corpora

Corpus B uses a different domain and generation model so results can be checked across corpora. Both corpora are fully synthetic; see their READMEs for provenance and limitations.

Corpus A ships in two halves, and both are ingested in order:

corpus_a/corpus/part_a/   2026-04-16 .. 2026-05-11
corpus_a/corpus/part_b/   2026-05-12 .. 2026-05-27   # supersedes part_a in places

The runner makes two ingest calls in order, testing whether memory handles later information that supersedes earlier claims.

What It Asks

Base set (155)

Hard set (31)

Added when the base set stopped separating systems.

as_of tests whether a system retained historical values instead of only the latest one.

All 186 questions are graded deterministically; no model runs in the scoring path. Answers are normalized for case, punctuation, whitespace, date spelling, and thousands separators before applying each mode's explicit rule set. Reports still include deferred_to_judge, which is always zero for shipped questions.

Abstention questions have no correct answer. Something merely scheduled has not happened; a merge request that exists only as a reserved URL has not been reviewed. Confidently describing either is scored wrong, and saying "the corpus does not say" is scored right.

Getting Started

Use Python 3.9+ on macOS or Linux, and install pytest and tesseract; the harness itself imports only the standard library. tesseract is required by two tests that read the README image and check it shows the run the suite verifies — without it those two fail rather than skip, because a check a machine might not run is not a check. A real run also needs an API key for the configured endpoint. The shipped adapters invoke python3, which a default Windows installation does not provide.

The entry point is python3 -m bench.runner and the offline check is python3 -m pytest tests/. Corpus A is the default; corpus_b/ holds the second corpus, and docs/SUBMITTING.md is the adapter contract.

1. Check it works — offline, free, no API key

Offline and free. This path sends nothing over the network.

cd examples/tools/agent-memory-benchmark
python3 -m pip install pytest
brew install tesseract        # macOS; on Linux: apt-get install tesseract-ocr
python3 -m pytest tests/      # expected: 234 passed

Then run the whole pipeline against a fixture whose score is known in advance:

python3 -m bench.runner \
    --adapter selftest/oracle \
    --corpus selftest/corpus \
    --questions selftest/questions.jsonl \
    --gold selftest/gold.jsonl

The oracle scores exactly 1.0; its sibling selftest/wrong scores exactly 0.0, wrong in a different way per grading mode. Any drift between the runner and the report moves a number the tests pin.

2. Score a real system

This spends money and sends data. Corpus text and questions are sent to the endpoint you point --upstream at, and you pay for those tokens. The harness itself writes only under results/ — but an adapter you add is not sandboxed and can write anywhere your user can.

export OPENAI_API_KEY=<your key>
python3 -m bench.runner --adapter adapters/naive_rag

The runner ingests both corpus halves in order, feeds corpus_a/questions/questions.jsonl to your system, grades the answers, and writes report.json + summary.md under results/runs/.

ledger_rag does not score the recipe itself. The adapter adds candidate selection and answer generation to a ledger designed for triage and ranking. It does not modify the recipe.

Reading A Result

Compare per-question-type accuracy, not only the overall average. Freshness separates cases with and without a competing stale claim. Evidence recall and citation coverage remain diagnostics rather than accuracy inputs.

Why the two axes are never combined, and what deterministic grading cannot express: docs/methodology.md.

Two reports are comparable only when their fingerprint blocks match — same corpus, same questions, same answer key, same normalization, same scorer.

Published Results

The repository includes one self-model run and one agentic retrieval run on corpus A with the same base model and grader. The artifacts show their scores and separate ingest and answer costs, but do not include the self-model's memory or enough accounting evidence for a cost ratio.

Read results/README.md before the table there. Those runs are corpus A only, on one base model; their answers predate a rename at publication; and their accounting cannot support a cost comparison. All four limits are stated where the numbers are.

Troubleshooting

Cleanup

Each run writes reports, verdicts, answers, and adapter state under results/runs/<timestamp>-<adapter>/. Delete that directory to reclaim space.

By default nothing outside that directory is modified by the harness. --out and --state will write wherever you point them, and an adapter is not sandboxed, so one you add can write elsewhere; see How the runner treats your adapter.

A generated run is ignored by Git. Publishing one requires an explicit .gitignore exception.

For Contributors

Everything below is for someone adding a system, changing the harness, or verifying a claim. A reader who only wants to run the benchmark can stop here.

Project Structure

agent-memory-benchmark/
├── bench/                       the harness — no third-party imports
│   ├── runner.py                orchestrates ingest → answer → grade → report
│   ├── grader.py                deterministic scoring; denial scope, normalization
│   ├── fingerprint.py           hashes corpus+questions+gold+scorer → comparability
│   ├── proxy.py                 local proxy that counts tokens crossing the wire
│   ├── normalize.py             case, punctuation and date-spelling normalization
│   ├── answer_contract.py       what an adapter is told about the answer format
│   ├── report.py                renders report.json into summary.md
│   └── pricing.py               model → USD table; unknown model ⇒ null, never a guess
├── corpus_a/                    corpus A — the default corpus
│   ├── README.md                what it is, and why the split matters
│   ├── corpus/                  425 documents in two dated halves
│   │   ├── part_a/              2026-04-16 .. 2026-05-11
│   │   ├── part_b/              2026-05-12 .. 2026-05-27 — supersedes part_a in places
│   │   ├── manifest.jsonl       per-document provenance
│   │   ├── counts.json          the document/message counts this README quotes
│   │   └── CANARY.txt           a string that must not appear in any model output
│   └── questions/               the questions and the answer key, kept out of corpus/
│       ├── questions.jsonl      the 186 questions asked of every system
│       ├── answers.jsonl        the answer key and its grading rules
│       └── factual_*.json       the drafting record — 103 kept, 17 dropped with a reason, 0 rejected
├── corpus_b/                    corpus B — a second domain, same shape as corpus_a/
├── adapters/                    one directory per system under test
│   ├── naive_rag/               ┐
│   ├── agentic_rag/             ├ shipped baselines, each with an adapter.json
│   ├── ledger_rag/              ┘
│   └── _lib/                    shared helpers adapters may import
├── selftest/                    six documents, six questions, two known-score adapters
│   ├── oracle/                  answers everything correctly — scores exactly 1.0
│   └── wrong/                   wrong in a different way per mode — scores exactly 0.0
├── results/                     published reference runs — see results/README.md
├── tools/regrade.py             re-score stored answers against the current rules
├── docs/                        SUBMITTING.md, methodology.md, provenance.md
└── tests/                       the benchmark's own tests

Dependency files: there are none to install for the harness — bench/ imports only the Python standard library. The offline check needs pytest (python3 -m pip install pytest) and tesseract (brew install tesseract, or apt-get install tesseract-ocr), the latter for the two tests that read the README image. An adapter for a third-party system declares its own requirements separately, inside its own directory.

Configuration

adapter.json — the contract for a system under test

Every adapter is one JSON file. name, ingest and answer are required — omitting any of them is an uncaught KeyError, not a default. model is optional despite naming the id a result is filed under: without it the runner falls back to the model it observed most.

{
  "name": "my-system",
  "model": "some-model-id",
  "accounting": "proxy",
  "ingest": ["my-cli", "ingest", "--corpus", "{corpus}", "--state", "{state}"],
  "answer": ["my-cli", "answer", "--state", "{state}"],
  "env": { "MY_SYSTEM_PYTHON": "python3" }
}

Field reference — the shape the runner validates:

interface AdapterConfig {
  /** Filed under this name in the report. */
  name: string;
  /** The model id a result is filed under. Not resolved by the harness. */
  model: string;
  /**
   * Argument arrays, never shell strings.
   * ingest receives {corpus} (the part directory), {state} and {part}.
   * answer receives {state} and {questions} — NOT {corpus}. Using {corpus}
   * there fails after both ingest calls have already run.
   */
  ingest: string[];
  answer: string[];
  /**
   * "proxy" (default) — calls go through the local proxy and are counted.
   * "local"           — a locally-hosted model; nothing crosses the wire, and
   *                     the run says so instead of reading as a cost of zero.
   */
  accounting?: "proxy" | "local";
  /** Defaults for adapter-specific variables. Anything already exported wins. */
  env?: Record<string, string>;
}

A declared accounting mode is enforced in both directions. Three things make a run invalid:

• it declares proxy and nothing crossed the proxy — the adapter either bypassed it or runs a local model and should say so;
• it declares local and requests crossed the proxy anyway — the declaration and the traffic disagree, and the traffic wins;
• a forwarded request came back without countable usage, because a cost nobody measured must not read as a cost of zero.

A local run that really is local stays valid, and stays out of cost comparisons. Only a fully counted run carries comparable_on_cost: true.

Environment variables

The shipped baselines default to gpt-4o and text-embedding-3-small, so they run as printed. Point --upstream at any OpenAI-compatible gateway; model ids are often namespaced differently there, so set model in the adapter to the id that gateway expects.

An adapter that needs a host-specific path declares it in its env block, and anything already exported wins — so nothing machine-specific has to be committed:

MY_SYSTEM_PYTHON=~/src/my-system/.venv/bin/python \
    python3 -m bench.runner --adapter adapters/my_system

⏱️ The timeout applies to each phase call, and ingest is called twice, so a default run can occupy up to eighteen hours before anything is killed.

Adding Your System

1. Write an adapter.json as above.
2. answer reads questions.jsonl on stdin and writes one JSON object per line to stdout:

{"id": "example-launch-date", "answer": "<your short answer>", "source_ids": ["E:2027-02-05T10-00-00__bbbb0002"]}

interface AnswerRow {
  /** Must match the question id it answers. */
  id: string;
  /**
   * A short, checkable answer. Blank, whitespace-only, non-string and missing
   * are all "nothing was said" — and one of them invalidates the whole run,
   * because an incomplete submission is not a lower score. A blank answer is
   * not credited as an abstention either.
   */
  answer: string;
  /** Optional. Buys the evidence diagnostics; omitting it costs nothing on accuracy. */
  source_ids?: string[];
}

1. Run it. Full instructions, including the path for a system that cannot be wrapped in a command line: docs/SUBMITTING.md.

How The Runner Treats Your Adapter

Adapters are not sandboxed. They run with your user, filesystem access, and environment credentials. Review an adapter before running it.

Commands are argument arrays, never shell strings. ["my-cli", "ingest", "--corpus", "{corpus}"] is executed directly with no shell in between, so a path containing a space or a quote cannot become a second command.

The guards prevent accidental answer-key access, not a malicious adapter. The benchmark root remains reachable through PYTHONPATH, as demonstrated by tests/test_isolation_is_not_a_sandbox.py. Trust the adapter behind any score. The guards provide:

• Adapters are launched from a scratch directory, not from the benchmark root, so a relative open("corpus_a/questions/answers.jsonl") finds nothing.
• Before each phase starts, the runner checks the phase was not handed a path it must not have. Ingest never receives the questions or the answer key; answer never receives the answer key. A run that would violate this stops before the adapter launches.
• Each adapter runs in its own process group, so a timeout — or a Ctrl-C — reaches the workers your adapter started, not just the adapter. A process that ignores SIGTERM is killed ten seconds later. Without this, a hung run leaves workers behind that keep the proxy open and keep spending tokens after the run is over.

Verification

Evidence level: local/static. Verified on Python 3.9.6, macOS. Nothing here exercises a live endpoint.

python3 -m pytest tests/     # expected: 234 passed

Expected result:

234 passed

This verifies the runner, grader, report renderer, ledger ingest, fixtures, and documented corpus counts. It does not test a live endpoint, real model call, or Windows.

Provenance And License

Authored and maintained by NVIDIA, contributed under Apache-2.0.

Both corpora are synthetic and use invented identities, projects, messages, and reserved .example domains. Nothing came from a live account.

How the corpora were generated, what was screened before publication, and the content hashes that say whether two scores are comparable: docs/provenance.md.

License: Apache-2.0 throughout — code, corpus, questions and answer key alike, under the repository's LICENSE.