In robot-assisted minimally invasive surgery, we introduce the Surgical Action Planning (SAP) task, which generates future action plans from visual inputs to address the absence of intraoperative predictive planning in current intelligent applications. SAP shows great potential for enhancing intraoperative guidance and automating procedures. However, it faces challenges such as understanding instrument-action relationships and tracking surgical progress. Large Language Models (LLMs) show promise in understanding surgical video content but remain underexplored for predictive decision-making in SAP, as they focus mainly on retrospective analysis. Challenges like data privacy, computational demands, and modality-specific constraints further highlight significant research gaps. To tackle these challenges, we introduce LLM-SAP, a Large Language Models-based Surgical Action Planning framework that predicts future actions and generates text responses by interpreting natural language prompts of surgical goals. The text responses potentially support surgical education, intraoperative decision-making, procedure documentation, and skill analysis. LLM-SAP integrates two novel modules: the Near-History Focus Memory Module (NHF-MM) for modeling historical states and the prompts factory for action planning. We evaluate LLM-SAP on our constructed CholecT50-SAP dataset using models like Qwen2.5 and Qwen2-VL, demonstrating its effectiveness in next-action prediction. Pre-trained LLMs are tested in a zero-shot setting, and supervised fine-tuning (SFT) with LoRA is implemented. Our experiments show that Qwen2.5-72B-SFT surpasses Qwen2.5-72B with a 19.3% higher accuracy.

翻译：在机器人辅助微创手术中，我们提出了手术动作规划任务，该任务从视觉输入生成未来动作计划，以解决当前智能应用中缺乏术中预测性规划的问题。手术动作规划在增强术中引导和实现流程自动化方面展现出巨大潜力。然而，其面临着理解器械-动作关系及追踪手术进程等挑战。大语言模型在理解手术视频内容方面表现出潜力，但由于其主要用于回顾性分析，在手术动作规划的预测性决策方面仍探索不足。数据隐私、计算需求及模态特定约束等挑战进一步凸显了显著的研究空白。为应对这些挑战，我们提出了LLM-SAP，一个基于大语言模型的手术动作规划框架，该框架通过解读手术目标的自然语言提示来预测未来动作并生成文本响应。这些文本响应可潜在支持手术教学、术中决策、流程记录与技能分析。LLM-SAP整合了两个新颖模块：用于建模历史状态的近历史聚焦记忆模块，以及用于动作规划的提示工厂。我们在自建的CholecT50-SAP数据集上使用Qwen2.5及Qwen2-VL等模型评估LLM-SAP，验证了其在下一动作预测方面的有效性。预训练大语言模型在零样本设置下进行测试，并采用LoRA进行监督微调。实验表明，经监督微调的Qwen2.5-72B模型相比基础版本实现了19.3%的准确率提升。