In this work, we leverage the intrinsic segmentation of language sequences and design a new positional encoding method called Bilevel Positional Encoding (BiPE). For each position, our BiPE blends an intra-segment encoding and an inter-segment encoding. The intra-segment encoding identifies the locations within a segment and helps the model capture the semantic information therein via absolute positional encoding. The inter-segment encoding specifies the segment index, models the relationships between segments, and aims to improve extrapolation capabilities via relative positional encoding. Theoretical analysis shows this disentanglement of positional information makes learning more effective. The empirical results also show that our BiPE has superior length extrapolation capabilities across a wide range of tasks in diverse text modalities.

翻译：本研究利用语言序列的固有分割特性，设计了一种名为双层级位置编码（Bilevel Positional Encoding, BiPE）的新型位置编码方法。针对每个位置，BiPE融合了段内编码与段间编码两种成分。其中，段内编码通过绝对位置编码标识段内位置，帮助模型捕获段内语义信息；段间编码则指定段落索引、建模段落间关系，并通过相对位置编码提升外推能力。理论分析表明，这种位置信息的解耦能提升学习效率。实验结果显示，BiPE在多种文本模态的广泛任务中均展现出卓越的长度外推能力。