In this paper, we propose a new event memory architecture (MemNet) for recurrent neural networks, which is universal for different types of time series data such as scalar, multivariate or symbolic. Unlike other external neural memory architectures, it stores key-value pairs, which separate the information for addressing and for content to improve the representation, as in the digital archetype. Moreover, the key-value pairs also avoid the compromise between memory depth and resolution that applies to memories constructed by the model state. One of the MemNet key characteristics is that it requires only linear adaptive mapping functions while implementing a nonlinear operation on the input data. MemNet architecture can be applied without modifications to scalar time series, logic operators on strings, and also to natural language processing, providing state-of-the-art results in all application domains such as the chaotic time series, the symbolic operation tasks, and the question-answering tasks (bAbI). Finally, controlled by five linear layers, MemNet requires a much smaller number of training parameters than other external memory networks as well as the transformer network. The space complexity of MemNet equals a single self-attention layer. It greatly improves the efficiency of the attention mechanism and opens the door for IoT applications.

翻译：本文提出了一种新的循环神经网络事件记忆架构（MemNet），该架构对不同类型的时序数据（如标量、多变量或符号数据）具有通用性。与其他外部神经记忆架构不同，它存储键值对，将用于寻址和内容表征的信息分离，类似于数字原型中的设计。此外，键值对还避免了由模型状态构造的记忆在记忆深度与分辨率之间的权衡。MemNet的关键特性之一是它仅需线性自适应映射函数，即可对输入数据实现非线性操作。MemNet架构无需修改即可应用于标量时间序列、字符串逻辑运算以及自然语言处理，在混沌时间序列、符号运算任务和问答任务（bAbI）等所有应用领域均取得最先进成果。最后，通过五个线性层控制，MemNet所需的训练参数数量远少于其他外部记忆网络以及Transformer网络。其空间复杂度与单层自注意力机制相当，显著提升了注意力机制的效率，为物联网应用开辟了道路。