Purpose: Surgical scene understanding plays a critical role in the technology stack of tomorrow's intervention-assisting systems in endoscopic surgeries. For this, tracking the endoscope pose is a key component, but remains challenging due to illumination conditions, deforming tissues and the breathing motion of organs. Method: We propose a solution for stereo endoscopes that estimates depth and optical flow to minimize two geometric losses for camera pose estimation. Most importantly, we introduce two learned adaptive per-pixel weight mappings that balance contributions according to the input image content. To do so, we train a Deep Declarative Network to take advantage of the expressiveness of deep-learning and the robustness of a novel geometric-based optimization approach. We validate our approach on the publicly available SCARED dataset and introduce a new in-vivo dataset, StereoMIS, which includes a wider spectrum of typically observed surgical settings. Results: Our method outperforms state-of-the-art methods on average and more importantly, in difficult scenarios where tissue deformations and breathing motion are visible. We observed that our proposed weight mappings attenuate the contribution of pixels on ambiguous regions of the images, such as deforming tissues. Conclusion: We demonstrate the effectiveness of our solution to robustly estimate the camera pose in challenging endoscopic surgical scenes. Our contributions can be used to improve related tasks like simultaneous localization and mapping (SLAM) or 3D reconstruction, therefore advancing surgical scene understanding in minimally-invasive surgery.

翻译：目的：手术场景理解在内窥镜手术的下一代辅助干预系统技术栈中扮演关键角色。其中，内窥镜位姿跟踪是核心组成部分，但因光照条件、组织变形及器官呼吸运动等因素而极具挑战性。方法：我们提出一种针对立体内窥镜的解决方案，通过估计深度与光流，最小化两个几何损失函数以实现相机位姿估计。尤为重要的是，我们引入两种经学习的自适应逐像素权重映射，可根据输入图像内容平衡各像素贡献。为此，我们训练深度声明式网络（Deep Declarative Network），以融合深度学习的表达能力与新型基于几何的优化方法的鲁棒性。我们在公开SCARED数据集上验证方法，并引入新体内数据集StereoMIS，其涵盖更广泛的手术场景典型观测。结果：本方法平均性能优于现有最先进方法，且在组织变形与呼吸运动显著的困难场景中优势更为突出。我们观察到提出的权重映射能衰减图像模糊区域（如变形组织）的像素贡献。结论：我们证明了解决方案在挑战性内窥镜手术场景中鲁棒估计相机位姿的有效性。该方法可改进同步定位与地图构建（SLAM）、三维重建等相关任务，从而推动微创手术中的手术场景理解技术进步。