Existing Large Language Models (LLMs) usually remain static after deployment, which might make it hard to inject new knowledge into the model. We aim to build models containing a considerable portion of self-updatable parameters, enabling the model to integrate new knowledge effectively and efficiently. To this end, we introduce MEMORYLLM, a model that comprises a transformer and a fixed-size memory pool within the latent space of the transformer. MEMORYLLM can self-update with text knowledge and memorize the knowledge injected earlier. Our evaluations demonstrate the ability of MEMORYLLM to effectively incorporate new knowledge, as evidenced by its performance on model editing benchmarks. Meanwhile, the model exhibits long-term information retention capacity, which is validated through our custom-designed evaluations and long-context benchmarks. MEMORYLLM also shows operational integrity without any sign of performance degradation even after nearly a million memory updates.

翻译：现有的基于Transformer的大语言模型（LLMs）通常在部署后保持静态，这可能使新知识难以注入模型。我们致力于构建包含大量可自我更新参数的模型，从而高效、有效地整合新知识。为此，我们提出MEMORYLLM——一种在Transformer隐空间中嵌入固定大小记忆池的模型。该模型能够利用文本知识进行自我更新，并记忆先前注入的知识。实验评估表明，MEMORYLLM能有效纳入新知识：其在模型编辑基准测试中的表现验证了这一能力。同时，模型展现出长期信息保留能力，这通过自定义评估与长上下文基准测试得到验证。即使在经历近百万次记忆更新后，MEMORYLLM仍保持操作完整性，未出现性能退化迹象。