End-to-end agent-memory benchmarks report a single hit@k per retriever, confounding lexical leakage (uncontrolled query/gold/distractor entity overlap) with tag-mixing (preferences, services, tools averaged together). We propose entity-collision, a system-agnostic protocol that pins the BM25 floor by construction -- every distractor shares the answer's entity tokens -- and stratifies queries by discriminator tag, so any lift over BM25 is attributable to the embedder. Applied to an open-source agent-memory testbed across 5 tags x 3 embedders x 5 collision degrees with paired-bootstrap 95% CIs, the protocol reveals a two-axis pattern: a 256-d hash trigram helps only on closed-vocabulary lexical tags at deep collision; MiniLM-384 dominates both axes; and a 2.7x-parameter BGE-large does not uniformly improve on MiniLM -- it wins on intent-style queries but loses on lexical ones. Encoder capacity alone is not the binding constraint. The synthetic intent-tag null replicates on LongMemEval (n=500) as a single-session-preference recall cliff. Adaptive vector-weight routing on LoCoMo is a measured null: 11.7pp of oracle headroom exists, but no signal we tested recovers it. All 26 result tables and 37 reproduce scripts are version-controlled and verified by a public registry; the protocol is exercised on a deterministically governed memory testbed (event-sourced decision log, DAG-state-machine schema lifecycle) so every reported CI is reproducible byte-for-byte from the ingest stream.
翻译:摘要:端到端智能体记忆基准测试报告了每个检索器单一的hit@k指标,这混淆了词汇泄露(未受控制的查询/黄金/干扰项实体重叠)与标签混合(偏好、服务、工具被平均混合)。我们提出实体碰撞协议,这是一种系统无关的协议,通过构造固定了BM25基线——每个干扰项共享答案的实体令牌——并按区分器标签对查询进行分层,因此任何超出BM25的提升均可归因于嵌入器。将该协议应用于跨5个标签×3个嵌入器×5个碰撞程度(含配对自举95%置信区间)的开源智能体记忆测试平台,揭示了一个双轴模式:256维哈希三元组仅在深度碰撞下的封闭词汇标签上有效;MiniLM-384在两个轴上均占主导地位;而参数规模2.7倍的BGE-large并未一致提升MiniLM——它在意图型查询上胜出,但在词汇型查询上表现不佳。编码器容量本身并非制约瓶颈。合成意图标签零假设结果在LongMemEval(n=500)上重现为单会话偏好召回悬崖,而LoCoMo上的自适应向量权重路由为测量的零假设:存在11.7个百分点的理想空间,但我们测试的所有信号均未能恢复。所有26个结果表和37个复现脚本均通过公共注册中心进行版本控制和验证;该协议在确定性驱动、事件溯源决策日志与有向无环图状态机模式生命周期管理的记忆测试平台上执行,因此每个报告的置信区间均可从摄取流逐字节复现。