Scaling up data, parameters, and test-time computation has been the mainstream methods to improve LLM systems (LLMsys), but their upper bounds are almost reached due to the gradual depletion of high-quality data and marginal gains obtained from larger computational resource consumption. Inspired by the abilities of human and traditional AI systems in learning from practice, constructing memory and continual learning frameworks for LLMsys has become an important and popular research direction in recent literature. Yet, existing benchmarks for LLM memory often focus on evaluating the system on homogeneous reading comprehension tasks with long-form inputs rather than testing their abilities to learn from accumulated user feedback in service time. Therefore, we propose a user feedback simulation framework and a comprehensive benchmark covering multiple domains, languages, and types of tasks to evaluate the continual learning abilities of LLMsys. Experiments show that the effectiveness and efficiency of state-of-the-art baselines are far from satisfying, and we hope this benchmark could pave the way for future studies on LLM memory and optimization algorithms. Website: https://memorybench.thuir.cn Code: https://github.com/THUIR/MemoryBench Data: https://huggingface.co/datasets/THUIR/MemoryBench Data-Full: https://huggingface.co/datasets/THUIR/MemoryBench-Full

翻译：扩大数据规模、参数量与测试时计算量是改进大语言系统（LLMsys）的主流方法，但由于高质量数据的逐渐枯竭以及更大计算资源消耗带来的边际收益递减，这些方法近乎达到上限。受人类和传统人工智能系统从实践中学习能力的启发，为LLMsys构建记忆与持续学习框架已成为近期文献中重要且热门的研究方向。然而，现有的大语言模型记忆基准测试往往聚焦于评估系统在长文本输入的同质化阅读理解任务上的表现，而非检验其通过服务过程中积累的用户反馈进行学习的能力。为此，我们提出一个用户反馈模拟框架及覆盖多领域、多语言、多任务类型的综合性基准测试，以评估LLMsys的持续学习能力。实验表明，当前最优基线方法的效果与效率远未令人满意，我们期望该基准能为未来大语言模型记忆与优化算法的研究铺平道路。网站：https://memorybench.thuir.cn 代码：https://github.com/THUIR/MemoryBench 数据：https://huggingface.co/datasets/THUIR/MemoryBench 完整数据：https://huggingface.co/datasets/THUIR/MemoryBench-Full