Pre-trained language models based on masked language modeling (MLM) excel in natural language understanding (NLU) tasks. While fine-tuned MLM-based encoders consistently outperform causal language modeling decoders of comparable size, recent decoder-only large language models (LLMs) perform on par with smaller MLM-based encoders. Although their performance improves with scale, LLMs fall short of achieving state-of-the-art results in information extraction (IE) tasks, many of which are formulated as sequence labeling (SL). We hypothesize that LLMs' poor SL performance stems from causal masking, which prevents the model from attending to tokens on the right of the current token. Yet, how exactly and to what extent LLMs' performance on SL can be improved remains unclear. We explore techniques for improving the SL performance of open LLMs on IE tasks by applying layer-wise removal of the causal mask (CM) during LLM fine-tuning. This approach yields performance gains competitive with state-of-the-art SL models, matching or outperforming the results of CM removal from all blocks. Our findings hold for diverse SL tasks, demonstrating that open LLMs with layer-dependent CM removal outperform strong MLM-based encoders and even instruction-tuned LLMs.

翻译：摘要：基于掩码语言建模（MLM）的预训练语言模型在自然语言理解（NLU）任务中表现出色。虽然微调后的MLM基编码器在性能上始终优于同规模的因果语言建模解码器，但近期仅解码器大型语言模型（LLM）的性能已与较小的MLM基编码器相当。尽管LLM的性能随规模扩大而提升，但它们在信息抽取（IE）任务中仍未能达到最优表现，其中许多任务被形式化为序列标注（SL）。我们假设LLM在SL上的较差表现源于因果掩码机制——该机制阻止模型关注当前词元右侧的上下文。然而，LLM在SL任务上的性能究竟能在多大程度上被改善，以及具体改善方式尚不明确。我们探索了在LLM微调过程中逐层移除因果掩码（CM）以提升其在IE任务中SL性能的技术。该方法取得了与当前最优SL模型相竞争的性能提升，其效果可与从所有模块中移除CM相媲美或更优。我们的发现适用于多种SL任务，表明采用层级依赖的CM移除策略的开放LLM优于强大的MLM基编码器，甚至超越经过指令微调的LLM。