English and Chinese, known as resource-rich languages, have witnessed the strong development of transformer-based language models for natural language processing tasks. Although Vietnam has approximately 100M people speaking Vietnamese, several pre-trained models, e.g., PhoBERT, ViBERT, and vELECTRA, performed well on general Vietnamese NLP tasks, including POS tagging and named entity recognition. These pre-trained language models are still limited to Vietnamese social media tasks. In this paper, we present the first monolingual pre-trained language model for Vietnamese social media texts, ViSoBERT, which is pre-trained on a large-scale corpus of high-quality and diverse Vietnamese social media texts using XLM-R architecture. Moreover, we explored our pre-trained model on five important natural language downstream tasks on Vietnamese social media texts: emotion recognition, hate speech detection, sentiment analysis, spam reviews detection, and hate speech spans detection. Our experiments demonstrate that ViSoBERT, with far fewer parameters, surpasses the previous state-of-the-art models on multiple Vietnamese social media tasks. Our ViSoBERT model is available\footnote{\url{https://huggingface.co/uitnlp/visobert}} only for research purposes.

翻译：英语和中文作为资源丰富的语言，在基于Transformer架构的自然语言处理任务的语言模型方面已取得强劲发展。尽管越南约有1亿母语者，但包括PhoBERT、ViBERT和vELECTRA在内的数个预训练模型已能良好处理通用越南语NLP任务（如词性标注和命名实体识别）。然而，这些预训练语言模型在越南社交媒体任务中仍存在局限性。本文提出了首个面向越南社交媒体文本的单语预训练语言模型ViSoBERT，该模型基于XLM-R架构，在高质量、多样化的越南社交媒体大规模语料库上完成预训练。我们进一步在越南社交媒体文本的五项重要自然语言下游任务中探索了该模型：情感识别、仇恨言论检测、情感分析、垃圾评论检测及仇恨言论片段检测。实验表明，ViSoBERT在参数量显著减少的情况下，在多个越南社交媒体任务上超越了此前最优模型。我们的ViSoBERT模型仅限研究用途开放获取\footnote{\url{https://huggingface.co/uitnlp/visobert}}。