LLM agents increasingly maintain long-term memory of user facts across sessions. Yet such memory is usually evaluated by aggregating accuracy over question rows or episodes. Because this approach scores question rows independently, even when several questions probe the same fact, it cannot show how that fact behaves as conditions change. We introduce MemTrace, a benchmark whose unit of measurement is the knowledge point: a single typed fact about the user, rather than an individual question. MemTrace probes each fact along three controlled dimensions: memory age, defined by how many sessions ago the fact appeared in the history; question type, covering current state, earlier state, and trajectory of change; and evidence condition, covering present, missing, and contradicted-by-false-premise settings. Evaluating 13 memory-system configurations across four paradigms, we find that similar pooled accuracy hides different failures: recovering a fact's current and earlier states does not imply tracking how it changed, and safe abstention does not imply correcting a false premise. The dominant bottleneck is evidence use, not retrieval: when systems fail, the evidence was retrievable 10 times more often than it was missing. These results suggest that improving long-term memory requires better use of reachable evidence, not simply more storage or retrieval.
翻译:LLM智能体在跨会话交互中持续维护用户事实的长期记忆。然而,此类记忆通常通过聚合问题行或片段上的准确率进行评估。由于这种评估方式独立计分每个问题行(即使多个问题探查同一事实),它无法揭示该事实在条件变化时的行为特征。我们提出MemTrace基准测试,其度量单元为知识要点:即关于用户的单一类型化事实(而非单个问题)。MemTrace沿着三个受控维度探查每个事实:记忆年龄(以事实在历史中出现的会话间隔定义)、问题类型(涵盖当前状态、先前状态及变化轨迹)以及证据条件(涵盖存在、缺失及被虚假前提驳斥三种设置)。通过评估横跨四种范式的13种记忆系统配置,我们发现相似的聚合准确率掩盖了不同的失效模式:恢复事实的当前状态与先前状态并不等价于追踪其变化轨迹,而安全弃权也不等价于纠正虚假前提。主要瓶颈在于证据利用而非检索:当系统失效时,可检索证据缺失的频率比证据不可用时高出10倍。这些结果表明,改进长期记忆需要更好地利用可获取证据,而非单纯增加存储容量或检索能力。