In recent years, pre-trained language models (PLMs) achieve the best performance on a wide range of natural language processing (NLP) tasks. While the first models were trained on general domain data, specialized ones have emerged to more effectively treat specific domains. In this paper, we propose an original study of PLMs in the medical domain on French language. We compare, for the first time, the performance of PLMs trained on both public data from the web and private data from healthcare establishments. We also evaluate different learning strategies on a set of biomedical tasks. In particular, we show that we can take advantage of already existing biomedical PLMs in a foreign language by further pre-train it on our targeted data. Finally, we release the first specialized PLMs for the biomedical field in French, called DrBERT, as well as the largest corpus of medical data under free license on which these models are trained.

翻译：近年来，预训练语言模型（PLMs）在各类自然语言处理（NLP）任务中取得了最佳表现。尽管早期模型基于通用领域数据训练，但为更有效处理特定领域，专用模型应运而生。本文针对法语医疗领域提出了一项原创性预训练语言模型研究。我们首次对比了基于网络公开数据与医疗机构私有数据训练的PLMs性能，并在多组生物医学任务中评估了不同学习策略。特别地，我们证明了可通过在目标数据上进一步预训练，利用已有外语生物医学PLMs的优势。最终，我们发布了首个面向法语生物医学领域的专用PLMs——DrBERT，以及这些模型所训练的规模最大的自由许可医学数据语料库。