Endoluminal surgery offers a minimally invasive option for early-stage gastrointestinal and urinary tract cancers but is limited by surgical tools and a steep learning curve. Robotic systems, particularly continuum robots, provide flexible instruments that enable precise tissue resection, potentially improving outcomes. This paper presents a visual perception platform for a continuum robotic system in endoluminal surgery. Our goal is to utilize monocular endoscopic image-based perception algorithms to identify position and orientation of flexible instruments and measure their distances from tissues. We introduce 2D and 3D learning-based perception algorithms and develop a physically-realistic simulator that models flexible instruments dynamics. This simulator generates realistic endoluminal scenes, enabling control of flexible robots and substantial data collection. Using a continuum robot prototype, we conducted module and system-level evaluations. Results show that our algorithms improve control of flexible instruments, reducing manipulation time by over 70% for trajectory-following tasks and enhancing understanding of surgical scenarios, leading to robust endoluminal surgeries.

翻译：腔内手术为早期胃肠道及泌尿系统癌症提供了一种微创治疗选择，但受限于手术工具及陡峭的学习曲线。机器人系统，特别是连续体机器人，提供了能够实现精确组织切除的柔性器械，有望改善手术效果。本文提出了一种用于腔内手术的连续体机器人系统视觉感知平台。我们的目标是利用基于单目内窥镜图像的感知算法，识别柔性器械的位置与姿态，并测量其与组织间的距离。我们引入了基于学习的二维与三维感知算法，并开发了一个物理逼真的模拟器，该模拟器对柔性器械的动力学进行建模。此模拟器能够生成逼真的腔内场景，从而实现对柔性机器人的控制并进行大规模数据采集。利用一个连续体机器人原型，我们进行了模块级和系统级评估。结果表明，我们的算法改进了对柔性器械的控制，在轨迹跟踪任务中将操作时间减少了70%以上，并增强了对手术场景的理解，从而实现稳健的腔内手术。