The increasing use of tools and solutions based on Large Language Models (LLMs) for various tasks in the medical domain has become a prominent trend. Their use in this highly critical and sensitive domain has thus raised important questions about their robustness, especially in response to variations in input, and the reliability of the generated outputs. This study addresses these questions by constructing a textual dataset based on the ICD-10-CM code descriptions, widely used in US hospitals and containing many clinical terms, and their easily reproducible rephrasing. We then benchmarked existing embedding models, either generalist or specialized in the clinical domain, in a semantic search task where the goal was to correctly match the rephrased text to the original description. Our results showed that generalist models performed better than clinical models, suggesting that existing clinical specialized models are more sensitive to small changes in input that confuse them. The highlighted problem of specialized models may be due to the fact that they have not been trained on sufficient data, and in particular on datasets that are not diverse enough to have a reliable global language understanding, which is still necessary for accurate handling of medical documents.

翻译：基于大语言模型（LLM）的工具和解决方案在医疗领域的应用日益广泛，已成为显著趋势。然而，这类工具在高度敏感且关键的医学领域的使用，引发了对其鲁棒性的重要质疑，尤其涉及输入变化时的响应稳定性以及生成输出的可靠性。本研究通过构建基于美国医院广泛使用的ICD-10-CM代码描述的文本数据集（该数据集包含大量临床术语及其易于复述的改写版本）来探讨上述问题。随后，我们在一项语义搜索任务中对现有嵌入模型（包括通用模型和临床领域专用模型）进行基准测试，目标是正确匹配改写文本与原描述。结果表明，通用模型的表现优于临床专用模型，提示现有临床专用模型对输入中的微小变化更为敏感，易导致混淆。专用模型表现不佳的原因可能在于其训练数据不足，尤其是缺乏足够多样化的数据集，因而未能建立可靠的全局语言理解能力——而这种能力对于准确处理医疗文档仍至关重要。