Despite the availability of computer-aided simulators and recorded videos of surgical procedures, junior residents still heavily rely on experts to answer their queries. However, expert surgeons are often overloaded with clinical and academic workloads and limit their time in answering. For this purpose, we develop a surgical question-answering system to facilitate robot-assisted surgical scene and activity understanding from recorded videos. Most of the existing VQA methods require an object detector and regions based feature extractor to extract visual features and fuse them with the embedded text of the question for answer generation. However, (1) surgical object detection model is scarce due to smaller datasets and lack of bounding box annotation; (2) current fusion strategy of heterogeneous modalities like text and image is naive; (3) the localized answering is missing, which is crucial in complex surgical scenarios. In this paper, we propose Visual Question Localized-Answering in Robotic Surgery (Surgical-VQLA) to localize the specific surgical area during the answer prediction. To deal with the fusion of the heterogeneous modalities, we design gated vision-language embedding (GVLE) to build input patches for the Language Vision Transformer (LViT) to predict the answer. To get localization, we add the detection head in parallel with the prediction head of the LViT. We also integrate GIoU loss to boost localization performance by preserving the accuracy of the question-answering model. We annotate two datasets of VQLA by utilizing publicly available surgical videos from MICCAI challenges EndoVis-17 and 18. Our validation results suggest that Surgical-VQLA can better understand the surgical scene and localize the specific area related to the question-answering. GVLE presents an efficient language-vision embedding technique by showing superior performance over the existing benchmarks.

翻译：尽管计算机辅助模拟器和手术录像已可供使用，初级住院医师仍高度依赖专家解答其疑问。然而，专家级外科医生常因临床和学术工作负荷过重而难以投入时间作答。为此，我们开发了一种手术问答系统，旨在从录制的视频中促进机器人辅助手术场景与活动理解。现有VQA方法大多需要借助目标检测器和基于区域的特征提取器来获取视觉特征，并将其与问题文本嵌入融合以生成答案。但存在以下问题：（1）手术目标检测模型因数据集较小且缺乏边界框标注而稀缺；（2）文本与图像等异质模态的融合策略尚不成熟；（3）缺乏在复杂手术场景中至关重要的定位性回答。本文提出面向机器人手术的可视化问答定位方法（Surgical-VQLA），在预测答案的同时定位特定手术区域。为处理异质模态融合，我们设计了门控视觉语言嵌入（GVLE）构建输入补丁，供语言视觉Transformer（LViT）预测答案。为实现定位，我们在LViT预测头旁并行添加检测头，同时结合GIoU损失函数，在保持问答模型精度的前提下提升定位性能。我们利用MICCAI挑战赛EndoVis-17和18的公开手术视频，标注了两个VQLA数据集。验证结果表明，Surgical-VQLA能更好地理解手术场景并定位与问答相关的特定区域。GVLE通过优于现有基准的性能，展现出高效的语言-视觉嵌入技术。