Since the emergence of the Transformer architecture, language model development has increased, driven by their promising potential. However, releasing these models into production requires properly understanding their behavior, particularly in sensitive domains such as medicine. Despite this need, the medical literature still lacks technical assessments of pre-trained language models, which are especially valuable in resource-constrained settings in terms of computational power or limited budget. To address this gap, we provide a comprehensive survey of language models in the medical domain. In addition, we selected a subset of these models for thorough evaluation, focusing on classification and text generation tasks. Our subset encompasses 53 models, ranging from 110 million to 13 billion parameters, spanning the three families of Transformer-based models and from diverse knowledge domains. This study employs a series of approaches for text classification together with zero-shot prompting instead of model training or fine-tuning, which closely resembles the limited resource setting in which many users of language models find themselves. Encouragingly, our findings reveal remarkable performance across various tasks and datasets, underscoring the latent potential of certain models to contain medical knowledge, even without domain specialization. Consequently, our study advocates for further exploration of model applications in medical contexts, particularly in resource-constrained settings. The code is available on https://github.com/anpoc/Language-models-in-medicine.

翻译：自Transformer架构问世以来，语言模型的发展因其广阔的应用前景而日益加速。然而，将这些模型投入实际应用需要准确理解其行为特征，尤其在医学等敏感领域。尽管存在这一需求，医学文献仍缺乏对预训练语言模型的技术性评估——这类评估在计算资源或预算受限的环境中具有特殊价值。为填补这一空白，我们对医学领域的语言模型进行了系统性综述。此外，我们从中选取了部分模型进行深入评估，重点关注分类与文本生成任务。所选子集涵盖53个模型，参数量从1.1亿到130亿不等，覆盖三大Transformer模型家族及多个知识领域。本研究采用系列文本分类方法配合零样本提示策略，而非传统的模型训练或微调方式，这更贴近多数语言模型使用者面临的资源受限场景。值得关注的是，研究结果揭示了这些模型在不同任务和数据集上的卓越表现，凸显了某些模型即使未经领域专门化训练仍蕴含医学知识的潜在能力。因此，本研究主张在医学场景中进一步探索模型应用，特别是在资源受限环境下。相关代码已发布于https://github.com/anpoc/Language-models-in-medicine。