The ubiquity of offensive content on social media is a growing cause for concern among companies and government organizations. Recently, transformer-based models such as BERT, XLNET, and XLM-R have achieved state-of-the-art performance in detecting various forms of offensive content (e.g. hate speech, cyberbullying, and cyberaggression). However, the majority of these models are limited in their capabilities due to their encoder-only architecture, which restricts the number and types of labels in downstream tasks. Addressing these limitations, this study presents the first pre-trained model with encoder-decoder architecture for offensive language identification with text-to-text transformers (T5) trained on two large offensive language identification datasets; SOLID and CCTK. We investigate the effectiveness of combining two datasets and selecting an optimal threshold in semi-supervised instances in SOLID in the T5 retraining step. Our pre-trained T5 model outperforms other transformer-based models fine-tuned for offensive language detection, such as fBERT and HateBERT, in multiple English benchmarks. Following a similar approach, we also train the first multilingual pre-trained model for offensive language identification using mT5 and evaluate its performance on a set of six different languages (German, Hindi, Korean, Marathi, Sinhala, and Spanish). The results demonstrate that this multilingual model achieves a new state-of-the-art on all the above datasets, showing its usefulness in multilingual scenarios. Our proposed T5-based models will be made freely available to the community.

翻译：社交媒体上攻击性内容的普遍存在日益引起企业和政府机构的担忧。近年来，基于Transformer的模型（如BERT、XLNET和XLM-R）在检测各类攻击性内容（如仇恨言论、网络霸凌和网络攻击）方面取得了最先进的性能。然而，这些模型大多受限于其仅含编码器的架构，从而限制了在下游任务中可使用的标签数量和类型。为解决这些局限，本研究首次提出了基于编码器-解码器架构的预训练模型，用于攻击性语言识别，该模型采用文本到文本的Transformer（T5），并在两个大型攻击性语言识别数据集（SOLID和CCTK）上进行训练。我们探究了在T5重训练步骤中，合并两个数据集以及为SOLID中的半监督实例选择最优阈值的有效性。我们的预训练T5模型在多个英语基准测试中优于其他针对攻击性语言检测微调的Transformer模型（如fBERT和HateBERT）。采用类似方法，我们还利用mT5训练了首个用于攻击性语言识别的多语言预训练模型，并在六种不同语言（德语、印地语、韩语、马拉地语、僧伽罗语和西班牙语）的数据集上评估其性能。结果表明，该多语言模型在所有上述数据集上均达到了新的最优水平，展现了其在多语言场景中的实用性。我们提出的基于T5的模型将免费向社区开放。