The Transformer architecture is crucial for numerous AI models, but it still faces challenges in long-range language modeling. Though several specific transformer architectures have been designed to tackle issues of long-range dependencies, existing methods like Transformer-XL are plagued by a high percentage of ineffective memories. In this study, we present a plug-and-play strategy, known as TRAining-free Memory Selection (TRAMS), that selects tokens participating in attention calculation based on one simple metric. This strategy allows us to keep tokens that are likely to have a high attention score with the current queries and ignore the other ones. We have tested our approach on the word-level benchmark (WikiText-103) and the character-level benchmark (enwik8), and the results indicate an improvement without having additional training or adding additional parameters.

翻译：Transformer架构对于众多人工智能模型至关重要，但在长程语言建模中仍面临挑战。尽管已有如Transformer-XL等特定架构被设计用于解决长程依赖问题，但现有方法中无效记忆的比例较高。本研究提出一种即插即用策略——免训练记忆选择（TRAMS），该策略基于单一度量标准选择参与注意力计算的词元。该方法能够保留与当前查询可能具有较高注意力分数的词元，同时忽略其他无关词元。我们在词级基准（WikiText-103）和字符级基准（enwik8）上进行了测试，结果表明该方法无需额外训练或添加参数即可实现性能提升。