In this work, we leverage the intrinsic segmentation of language sequences and design a new positional encoding method called Bilevel Positional Encoding (BiPE). For each position, our BiPE blends an intra-segment encoding and an inter-segment encoding. The intra-segment encoding identifies the locations within a segment and helps the model capture the semantic information therein via absolute positional encoding. The inter-segment encoding specifies the segment index, models the relationships between segments, and aims to improve extrapolation capabilities via relative positional encoding. Theoretical analysis shows this disentanglement of positional information makes learning more effective. The empirical results also show that our BiPE has superior length extrapolation capabilities across a wide range of tasks in diverse text modalities.

翻译：在本工作中，我们利用语言序列固有的分段特性，设计了一种称为双层位置编码的新位置编码方法。对于每个位置，我们的BiPE融合了段内编码和段间编码。段内编码识别段内的位置，并通过绝对位置编码帮助模型捕捉其中的语义信息。段间编码指定段索引，建模段间关系，旨在通过相对位置编码提升外推能力。理论分析表明，这种位置信息的解耦使学习更有效。实证结果也显示，我们的BiPE在多种文本模态的广泛任务中具有卓越的长度外推能力。