Sequence labeling (SL) tasks, where labels are assigned to tokens, are abundant in NLP (e.g., named entity recognition and aspect-based sentiment analysis). Owing to the intuition that they require bidirectional context, SL tasks are commonly tackled with encoder-only models. Recent work also shows that removing the causal mask in fine-tuning enables decoder-based LLMs to become effective token classifiers. Less work, however, focused on (supervised) generative SL, a more natural setting for causal LLMs. Due to their rapid scaling, causal LLMs applied to SL are expected to outperform encoders, whose own development has stagnated. In this work, we propose supervised in-context fine-tuning (SIFT) for generative SL. SIFT casts SL tasks as constrained response generation, natural to LLMs, combining in-context learning (ICL) from demonstrations with supervised fine-tuning. SIFT considerably outperforms both ICL and decoder-as-encoder fine-tuning baselines on a range of standard SL tasks. We further find that although long context hinders the performance of generative SL in both ICL and SIFT, this deficiency can be mitigated by removing the instruction, as instructions are shown to be largely unnecessary for achieving strong SL performance with SIFT. Our findings highlight strengths and limitations of SL with LLMs, underscoring the importance of a response-based generative task formulation for effective SL performance.

翻译：序列标注（SL）任务（即为词元分配标签的任务）在自然语言处理中十分常见（例如命名实体识别和基于方面的情感分析）。由于直觉上认为此类任务需要双向上下文，通常采用仅编码器模型进行处理。近期研究也表明，在微调过程中移除因果掩码可使基于解码器的大型语言模型成为有效的词元分类器。然而，较少研究关注（监督式）生成式序列标注——这对因果性大型语言模型而言是更自然的设定。得益于其快速的规模扩展，应用于序列标注的因果性大型语言模型有望超越编码器模型，而后者的自身发展已趋于停滞。本文提出用于生成式序列标注的监督式上下文微调方法。该方法将序列标注任务转化为受约束的响应生成问题——这对大型语言模型而言更为自然，同时结合了基于示例的上下文学习与监督式微调。在一系列标准序列标注任务上，该方法显著超越了上下文学习及解码器即编码器微调基线。我们进一步发现，虽然长上下文会损害生成式序列标注在上下文学习及监督式上下文微调中的性能，但通过移除指令可缓解此缺陷，因为研究表明在使用监督式上下文微调实现强大序列标注性能时，指令在很大程度上并非必需。我们的发现揭示了大型语言模型处理序列标注任务的优势与局限，强调了基于响应的生成式任务表述对于实现高效序列标注性能的重要性。