This paper presents the PALI team's winning system for SemEval-2021 Task 2: Multilingual and Cross-lingual Word-in-Context Disambiguation. We fine-tune XLM-RoBERTa model to solve the task of word in context disambiguation, i.e., to determine whether the target word in the two contexts contains the same meaning or not. In the implementation, we first specifically design an input tag to emphasize the target word in the contexts. Second, we construct a new vector on the fine-tuned embeddings from XLM-RoBERTa and feed it to a fully-connected network to output the probability of whether the target word in the context has the same meaning or not. The new vector is attained by concatenating the embedding of the [CLS] token and the embeddings of the target word in the contexts. In training, we explore several tricks, such as the Ranger optimizer, data augmentation, and adversarial training, to improve the model prediction. Consequently, we attain first place in all four cross-lingual tasks.

翻译：本文展示了 PALI 团队在 SemEval 2021 任务2: 多语种和跨语言 Word- in-Context Disfendation 2: 我们微调 XLM- ROBERTA 模式, 以在背景脱节中解决单词任务, 即确定两个背景下的目标单词是否包含相同的含义。 在执行过程中, 我们首先专门设计一个输入标签, 以强调目标字。 其次, 我们在 XLM- ROBERTA 的精细调整嵌入上安装一个新的矢量, 并将其输入一个完全连接的网络, 以输出上下文中的目标单词是否具有相同含义的概率。 新的矢量是通过将[ CLS] 符号和目标单词嵌入到背景中来实现的。 在培训中, 我们探索了几个技巧, 如 游侠优化器、 数据增强和对抗性训练, 来改进模型预测。 因此, 我们在所有四种跨语言任务中获得了第一位 。