Modern language models (LMs) are trained in an autoregressive manner, conditioned only on the prefix. In contrast, sequence labeling (SL) tasks assign labels to each individual input token, naturally benefiting from bidirectional context. This discrepancy has historically led SL to rely on inherently bidirectional encoder-only models. However, the rapid development of decoder-only models has raised the question of whether they can be adapted to SL. While causal mask removal has emerged as a viable technique for adapting decoder-only models to leverage the full context for SL, it requires considerable changes to the base model functionality. In this work, we explore sequence repetition (SR) as a less invasive alternative for enabling bidirectionality in decoder-only models. Through fine-tuning experiments, we show that SR inherently makes decoders bidirectional, improving the quality of token-level embeddings and surpassing encoders and unmasked decoders. Contrary to earlier claims, we find that increasing the number of repetitions does not degrade SL performance. Finally, we demonstrate that embeddings from intermediate layers are highly effective for SR, comparable to those from final layers, while being significantly more efficient to compute. Our findings underscore that SR alleviates the structural limitations of decoders, enabling more efficient and adaptable LMs and broadening their applicability to other token-level tasks.

翻译：现代语言模型（LMs）以自回归方式进行训练，仅以前缀为条件。相比之下，序列标注（SL）任务为每个输入词元分配标签，天然受益于双向上下文。这种差异历来导致SL依赖于本质上是双向的仅编码器模型。然而，仅解码器模型的快速发展引发了其能否适应SL任务的疑问。虽然因果掩码移除已成为一种可行的技术，用于使仅解码器模型适应SL任务以利用完整上下文，但这需要对基础模型功能进行大量修改。在本工作中，我们探索序列重复（SR）作为一种侵入性更小的替代方案，以实现仅解码器模型的双向性。通过微调实验，我们证明SR本质上使解码器具有双向性，提升了词元级嵌入的质量，并超越了编码器及无掩码解码器。与早期观点相反，我们发现增加重复次数并不会降低SL性能。最后，我们证明了中间层的嵌入对于SR极为有效，其性能与最终层嵌入相当，同时计算效率显著更高。我们的研究结果强调，SR缓解了解码器的结构限制，使得语言模型更高效、更适应性强，并拓宽了其适用于其他词元级任务的范围。