The integration of Large Language Models (LLMs) into medical applications has sparked widespread interest across the healthcare industry, from drug discovery and development to clinical decision support, assisting telemedicine, medical devices, and healthcare insurance applications. This perspective paper aims to discuss the inner workings of building LLM-powered medical AI applications and introduces a comprehensive framework for their development. We review existing literature and outline the unique challenges of applying LLMs in specialized medical contexts. Additionally, we introduce a three-step framework to organize medical LLM research activities: 1) Modeling: breaking down complex medical workflows into manageable steps for developing medical-specific models; 2) Optimization: optimizing the model performance with crafted prompts and integrating external knowledge and tools, and 3) System engineering: decomposing complex tasks into subtasks and leveraging human expertise for building medical AI applications. Furthermore, we offer a detailed use case playbook that describes various LLM-powered medical AI applications, such as optimizing clinical trial design, enhancing clinical decision support, and advancing medical imaging analysis. Finally, we discuss various challenges and considerations for building medical AI applications with LLMs, such as handling hallucination issues, data ownership and compliance, privacy, intellectual property considerations, compute cost, sustainability issues, and responsible AI requirements.

翻译：大型语言模型（LLM）在医疗应用中的集成已引发医疗健康行业的广泛关注，其应用范围涵盖从药物发现与开发、临床决策支持，到辅助远程医疗、医疗设备及健康保险应用等多个领域。本视角论文旨在探讨构建基于LLM的医疗AI应用的内在机制，并提出其开发的综合框架。我们回顾现有文献，并概述了在专业医疗场景中应用LLM所面临的独特挑战。此外，我们引入一个三阶段框架来组织医疗LLM研究活动：1）建模：将复杂医疗工作流分解为可管理的步骤，以开发医疗专用模型；2）优化：通过精心设计的提示词优化模型性能，并整合外部知识与工具；3）系统工程：将复杂任务分解为子任务，并利用人类专业知识构建医疗AI应用。进一步地，我们提供了一份详细的使用案例手册，描述了多种基于LLM的医疗AI应用，例如优化临床试验设计、增强临床决策支持以及推进医学影像分析。最后，我们讨论了利用LLM构建医疗AI应用时面临的各种挑战与考量，包括处理幻觉问题、数据所有权与合规性、隐私、知识产权考量、计算成本、可持续性问题以及负责任AI的要求。