How humans and machines make sense of current inputs for relation reasoning and question-answering while putting the perceived information into context of our past memories, has been a challenging conundrum in cognitive science and artificial intelligence. Inspired by human brain's memory system and cognitive architectures, we propose a PMI framework that consists of perception, memory and inference components. Notably, the memory module comprises working and long-term memory, with the latter endowed with a higher-order structure to retain extensive and complex relational knowledge and experience. Through a differentiable competitive write access, current perceptions update working memory, which is later merged with long-term memory via outer product associations, reducing information conflicts and averting memory overflow. In the inference module, relevant information is retrieved from two separate memory origins and associatively integrated to attain a more comprehensive and precise interpretation of current perceptions. We exploratively apply our PMI to improve prevailing Transformers and CNN models on question-answering tasks like bAbI-20k and Sort-of-CLEVR datasets, as well as detecting equilateral triangles, language modeling and image classification tasks, and in each case, our PMI enhancements consistently outshine their original counterparts significantly. Visualization analyses reveal that relational memory consolidation, along with the interaction and integration of information from diverse memory sources, substantially contributes to the model effectiveness on inference tasks.

翻译：人类和机器如何理解当前输入以进行关系推理和问答，同时将感知信息置于过去记忆的上下文中，一直是认知科学和人工智能领域的棘手难题。受人类大脑记忆系统与认知架构的启发，我们提出了一个包含感知、记忆和推理组件的PMI框架。其中，记忆模块包含工作记忆和长期记忆，后者具备高阶结构以存储广泛而复杂的关系知识和经验。通过可微分的竞争性写入机制，当前感知内容更新工作记忆，随后通过外积关联与长期记忆融合，减少信息冲突并避免记忆溢出。在推理模块中，分别从两种不同的记忆来源检索相关信息，并通过关联整合实现对当前感知更全面、精确的解读。我们探索性地将PMI应用于改进主流Transformer和CNN模型，在bAbI-20k和Sort-of-CLEVR等数据集上的问答任务、等边三角形检测、语言建模以及图像分类任务中，PMI增强模型均显著优于原始模型。可视化分析表明，关系记忆的巩固以及不同记忆源信息的交互与整合，对模型在推理任务上的有效性起到了关键作用。