The attention mechanism is a critical component of Large Language Models (LLMs) that allows tokens in a sequence to interact with each other, but is order-invariant. Incorporating position encoding (PE) makes it possible to address by position, such as attending to the i-th token. However, current PE methods use token counts to derive position, and thus cannot generalize to higher levels of abstraction, such as attending to the i-th sentence. In this paper, we propose a new position encoding method, Contextual Position Encoding (CoPE), that allows positions to be conditioned on context by incrementing position only on certain tokens determined by the model. This allows more general position addressing such as attending to the $i$-th particular word, noun, or sentence. We show that CoPE can solve the selective copy, counting and Flip-Flop tasks where popular position embeddings fail, and improves perplexity on language modeling and coding tasks.

翻译：注意力机制是大语言模型（LLM）的关键组件，它允许序列中的标记相互交互，但其本身是顺序无关的。引入位置编码（PE）使其能够通过位置进行寻址，例如关注第i个标记。然而，当前的位置编码方法使用标记计数来推导位置，因此无法推广到更高层次的抽象，例如关注第i个句子。本文提出了一种新的位置编码方法——上下文位置编码（CoPE），该方法通过仅对模型确定的特定标记递增位置，使位置能够以上下文为条件。这使得更通用的位置寻址成为可能，例如关注第$i$个特定单词、名词或句子。我们证明，CoPE能够解决流行的位置嵌入方法所无法处理的选择性复制、计数和Flip-Flop任务，并在语言建模和代码任务上降低了困惑度。