Even though Transformers are extensively used for Natural Language Processing tasks, especially for machine translation, they lack an explicit memory to store key concepts of processed texts. This paper explores the properties of the content of symbolic working memory added to the Transformer model decoder. Such working memory enhances the quality of model predictions in machine translation task and works as a neural-symbolic representation of information that is important for the model to make correct translations. The study of memory content revealed that translated text keywords are stored in the working memory, pointing to the relevance of memory content to the processed text. Also, the diversity of tokens and parts of speech stored in memory correlates with the complexity of the corpora for machine translation task.

翻译：尽管Transformer被广泛用于自然语言处理任务（尤其是机器翻译），但其缺乏显式记忆来存储已处理文本的关键概念。本文探讨了在Transformer模型解码器中添加符号工作记忆的内容特性。此类工作记忆能够提升机器翻译任务中模型预测的质量，并作为对模型实现正确翻译至关重要的信息的神经符号表征。对记忆内容的研究表明，翻译文本的关键词被存储于工作记忆中，这指向了记忆内容与已处理文本的相关性。此外，存储在记忆中的词符和词性的多样性与机器翻译任务语料库的复杂度存在相关性。