We introduce a dynamic benchmarking system for conversational agents that evaluates their performance through a single, simulated, and lengthy user$\leftrightarrow$agent interaction. The interaction is a conversation between the user and agent, where multiple tasks are introduced and then undertaken concurrently. We context switch regularly to interleave the tasks, which constructs a realistic testing scenario in which we assess the Long-Term Memory, Continual Learning, and Information Integration capabilities of the agents. Results from both proprietary and open-source Large-Language Models show that LLMs in general perform well on single-task interactions, but they struggle on the same tasks when they are interleaved. Notably, short-context LLMs supplemented with an LTM system perform as well as or better than those with larger contexts. Our benchmark suggests that there are other challenges for LLMs responding to more natural interactions that contemporary benchmarks have heretofore not been able to capture.

翻译：我们提出了一种用于对话智能体的动态基准测试系统，该系统通过单一、模拟且冗长的用户$\leftrightarrow$智能体交互来评估其性能。该交互是用户与智能体之间的对话，其中引入多个任务并随后并发执行。我们定期进行上下文切换以交错执行这些任务，从而构建了一个现实的测试场景，用以评估智能体的长期记忆、持续学习和信息整合能力。来自专有和开源大型语言模型的结果表明，LLMs 通常在单任务交互中表现良好，但在相同任务交错执行时则表现不佳。值得注意的是，配备了长期记忆系统的短上下文 LLMs 的表现与具有更大上下文的模型相当甚至更优。我们的基准测试表明，对于 LLMs 响应更自然的交互，还存在一些当代基准测试迄今未能捕捉到的挑战。