Prompt-based methods have been successfully applied to multilingual pretrained language models for zero-shot cross-lingual understanding. However, most previous studies primarily focused on sentence-level classification tasks, and only a few considered token-level labeling tasks such as Named Entity Recognition (NER) and Part-of-Speech (POS) tagging. In this paper, we propose Token-Level Prompt Decomposition (ToPro), which facilitates the prompt-based method for token-level sequence labeling tasks. The ToPro method decomposes an input sentence into single tokens and applies one prompt template to each token. Our experiments on multilingual NER and POS tagging datasets demonstrate that ToPro-based fine-tuning outperforms Vanilla fine-tuning and Prompt-Tuning in zero-shot cross-lingual transfer, especially for languages that are typologically different from the source language English. Our method also attains state-of-the-art performance when employed with the mT5 model. Besides, our exploratory study in multilingual large language models shows that ToPro performs much better than the current in-context learning method. Overall, the performance improvements show that ToPro could potentially serve as a novel and simple benchmarking method for sequence labeling tasks.

翻译：基于提示的方法已成功应用于多语言预训练语言模型，以实现零样本跨语言理解。然而，大多数先前研究主要关注句子级分类任务，仅有少数考虑了令牌级标注任务，如命名实体识别（NER）和词性标注（POS）。本文提出令牌级提示分解（ToPro），以促进基于提示的方法在令牌级序列标注任务中的应用。ToPro方法将输入句子分解为单个令牌，并对每个令牌应用一个提示模板。我们在多语言NER和POS标注数据集上的实验表明，在零样本跨语言迁移中，基于ToPro的微调优于普通微调和提示微调，尤其是对于与源语言英语在类型学上差异较大的语言。当与mT5模型结合使用时，我们的方法还达到了最先进的性能。此外，我们对多语言大语言模型的探索性研究表明，ToPro的表现远优于当前的上下文学习方法。总体而言，性能提升表明ToPro有望作为一种新颖且简单的序列标注任务基准方法。