One of the recent advances in surgical AI is the recognition of surgical activities as triplets of (instrument, verb, target). Albeit providing detailed information for computer-assisted intervention, current triplet recognition approaches rely only on single frame features. Exploiting the temporal cues from earlier frames would improve the recognition of surgical action triplets from videos. In this paper, we propose Rendezvous in Time (RiT) - a deep learning model that extends the state-of-the-art model, Rendezvous, with temporal modeling. Focusing more on the verbs, our RiT explores the connectedness of current and past frames to learn temporal attention-based features for enhanced triplet recognition. We validate our proposal on the challenging surgical triplet dataset, CholecT45, demonstrating an improved recognition of the verb and triplet along with other interactions involving the verb such as (instrument, verb). Qualitative results show that the RiT produces smoother predictions for most triplet instances than the state-of-the-arts. We present a novel attention-based approach that leverages the temporal fusion of video frames to model the evolution of surgical actions and exploit their benefits for surgical triplet recognition.

翻译：近年来，手术人工智能领域的一项重要进展是将手术活动识别为（器械、动词、目标）的三元组。尽管为计算机辅助干预提供了详细信息，当前的三元组识别方法仅依赖于单帧特征。利用早期帧中的时间线索将改善视频中手术动作三元组的识别。在本文中，我们提出“随时间相会”（RiT）——一种通过时间建模扩展当前最优模型Rendezvous的深度学习模型。我们的RiT更关注动词，探索当前帧与过去帧之间的关联性，以学习基于注意力的时间特征，从而增强三元组识别。我们在具有挑战性的手术三元组数据集CholecT45上验证了我们的提议，结果表明动词和三元组的识别性能得到提升，同时涉及动词的其他交互关系（如（器械、动词））也得到改善。定性结果显示，与现有最优方法相比，RiT对大多数三元组实例能产生更平滑的预测。我们提出了一种新颖的基于注意力的方法，通过视频帧的时间融合来建模手术动作的演化，并利用其优势进行手术三元组识别。