The Attention module finds common usage in language modeling, presenting distinct challenges within the broader scope of Natural Language Processing. Multi-Head Attention (MHA) employs an absolute positional encoding, which imposes limitations on token length and entails substantial memory consumption during the processing of embedded inputs. The current remedy proposed by researchers involves the utilization of relative positional encoding, similar to the approach adopted in Transformer-XL or Relative Multi-Head Attention (RMHA), albeit the employed architecture consumes considerable memory resources. To address these challenges, this study endeavors to refine MHA, leveraging relative positional encoding in conjunction with the Depth-Wise Convolutional Layer architecture, which promises heightened accuracy coupled with minimized memory usage. The proposed RCMHA framework entails the modification of two integral components: firstly, the application of the Depth-Wise Convolutional Layer to the input embedding, encompassing Query, Key, and Value parameters; secondly, the incorporation of Relative Positional Encoding into the attention scoring phase, harmoniously integrated with Scaled Dot-Product Attention. Empirical experiments underscore the advantages of RCMHA, wherein it exhibits superior accuracy, boasting a score of 0.572 in comparison to alternative attention modules such as MHA, Multi-DConv-Head Attention (MDHA), and RMHA. Concerning memory utilization, RMHA emerges as the most frugal, demonstrating an average consumption of 2.98 GB, surpassing RMHA which necessitates 3.5 GB.

翻译：注意力模块在语言建模中广泛应用，并在自然语言处理的更广泛领域呈现出独特挑战。多头注意力（MHA）采用绝对位置编码，这限制了令牌长度，并在处理嵌入输入时消耗大量内存。研究者当前提出的解决方案是采用相对位置编码，类似于Transformer-XL或相对多头注意力（RMHA）中的方法，但所用架构仍消耗大量内存资源。为解决这些问题，本研究致力于改进MHA，利用相对位置编码结合深度可分离卷积层架构，以期在最小化内存使用的同时提高精度。所提出的RCMHA框架涉及两个核心组件的改进：首先，将深度可分离卷积层应用于包含查询、键和值参数的输入嵌入；其次，在注意力评分阶段引入相对位置编码，并与缩放点积注意力无缝集成。实验结果表明，RCMHA展现出显著优势，其准确性得分为0.572，优于MHA、多头深度可分离卷积注意力（MDHA）和RMHA等其他注意力模块。在内存使用方面，RMHA最为节省，平均消耗2.98 GB，而RCMHA需3.5 GB。