dolphinbench

Mapping the Pareto frontier of agent memory.

DolphinBench evaluates whether an agent uses long-term memory to take the right actions on a user's behalf. It contains three simulated users, approximately 500,000 tokens of history per user, and 600 tool-using tests across their work and personal lives.

A test asks the agent to complete a task using information from earlier conversations. Remembering that information can determine a recipient, a date, a tool argument, or the content of an email or document. The benchmark grades what the agent actually does through simulated apps, not just what it says it remembers.

Use DolphinBench with your own agent harness, model, and memory implementation. The repository includes the dataset, simulated apps, runner, graders, and the reference ingestion and evaluation code for the paper's configurations.

Browse the official runs for each agent's costs, ingestion records, test conversations, tool calls, and grades.

How the benchmark works

• Ingest the history. Process the original dated messages in order through

your agent and its normal memory-writing behavior. Keep a separate memory store for each persona and configuration.

• Preserve the completed memory. Wait for memory processing to finish and

record the completed checkpoint.

• Run the tests. Start a fresh conversation and fresh simulated app state