Recent advances in NLP have significantly improved the performance of language models on a variety of tasks. While these advances are largely driven by the availability of large amounts of data and computational power, they also benefit from the development of better training methods and architectures. In this paper, we introduce CamemBERTa, a French DeBERTa model that builds upon the DeBERTaV3 architecture and training objective. We evaluate our model's performance on a variety of French downstream tasks and datasets, including question answering, part-of-speech tagging, dependency parsing, named entity recognition, and the FLUE benchmark, and compare against CamemBERT, the state-of-the-art monolingual model for French. Our results show that, given the same amount of training tokens, our model outperforms BERT-based models trained with MLM on most tasks. Furthermore, our new model reaches similar or superior performance on downstream tasks compared to CamemBERT, despite being trained on only 30% of its total number of input tokens. In addition to our experimental results, we also publicly release the weights and code implementation of CamemBERTa, making it the first publicly available DeBERTaV3 model outside of the original paper and the first openly available implementation of a DeBERTaV3 training objective. https://gitlab.inria.fr/almanach/CamemBERTa

翻译：自然语言处理的最新进展显著提升了语言模型在各类任务上的性能。尽管这些进步主要得益于大量可用数据与计算资源，但更优的训练方法与架构发展同样起到了关键作用。本文提出CamemBERTa——一个基于DeBERTaV3架构与训练目标的法语DeBERTa模型。我们在多项法语下游任务与数据集上评估模型性能，包括问答、词性标注、依存句法分析、命名实体识别及FLUE基准测试，并与当前法语单语最优模型CamemBERT进行对比。实验结果表明：在相同训练token数量下，我们的模型在多数任务上优于采用MLM训练的BERT类模型。此外，尽管新模型仅使用CamemBERT总输入token数的30%进行训练，其在下游任务上仍能达到相似或更优表现。除实验结果外，我们还公开发布了CamemBERTa的权重与代码实现，使其成为原始论文之外首个公开可用的DeBERTaV3模型，以及首个公开实现的DeBERTaV3训练目标。https://gitlab.inria.fr/almanach/CamemBERTa