This paper presents medBERT.de, a pre-trained German BERT model specifically designed for the German medical domain. The model has been trained on a large corpus of 4.7 Million German medical documents and has been shown to achieve new state-of-the-art performance on eight different medical benchmarks covering a wide range of disciplines and medical document types. In addition to evaluating the overall performance of the model, this paper also conducts a more in-depth analysis of its capabilities. We investigate the impact of data deduplication on the model's performance, as well as the potential benefits of using more efficient tokenization methods. Our results indicate that domain-specific models such as medBERT.de are particularly useful for longer texts, and that deduplication of training data does not necessarily lead to improved performance. Furthermore, we found that efficient tokenization plays only a minor role in improving model performance, and attribute most of the improved performance to the large amount of training data. To encourage further research, the pre-trained model weights and new benchmarks based on radiological data are made publicly available for use by the scientific community.

翻译：本文提出了medBERT.de，一个专门为德语医学领域预训练的BERT模型。该模型基于包含470万份德语医学文档的大规模语料库进行训练，在涵盖广泛学科和医学文档类型的八项不同医学基准测试中，取得了最新的最优性能。除了评估模型的整体性能外，本文还对其能力进行了更深入的分析。我们研究了数据去重对模型性能的影响，以及使用更高效分词方法的潜在优势。结果表明，medBERT.de等域特定模型尤其适用于较长的文本，且训练数据去重不一定会带来性能提升。此外，我们发现高效分词在提升模型性能方面作用甚微，并将大部分性能提升归因于大规模训练数据。为推动进一步研究，我们公开了预训练模型的权重以及基于放射学数据构建的新基准测试，供科学界使用。