In addition to generating data and annotations, devising sensible data splitting strategies and evaluation metrics is essential for the creation of a benchmark dataset. This practice ensures consensus on the usage of the data, homogeneous assessment, and uniform comparison of research methods on the dataset. This study focuses on CholecT50, which is a 50 video surgical dataset that formalizes surgical activities as triplets of <instrument, verb, target>. In this paper, we introduce the standard splits for the CholecT50 and CholecT45 datasets and show how they compare with existing use of the dataset. CholecT45 is the first public release of 45 videos of CholecT50 dataset. We also develop a metrics library, ivtmetrics, for model evaluation on surgical triplets. Furthermore, we conduct a benchmark study by reproducing baseline methods in the most predominantly used deep learning frameworks (PyTorch and TensorFlow) to evaluate them using the proposed data splits and metrics and release them publicly to support future research. The proposed data splits and evaluation metrics will enable global tracking of research progress on the dataset and facilitate optimal model selection for further deployment.

翻译：除生成数据和标注外，设计合理的数据划分策略与评估指标对于构建基准数据集至关重要。这一实践可确保数据使用规范、评估标准统一，并实现数据集中研究方法的可比性评估。本研究聚焦于CholecT50——一个包含50段手术视频的数据集，将手术活动形式化为<器械、操作、目标>三元组。本文介绍了CholecT50和CholecT45数据集的标准划分方案，并展示了其与现有数据集使用方式的差异。CholecT45是CholecT50数据集首次公开发布的45段视频子集。我们还开发了面向手术三元组模型评估的指标库ivtmetrics。通过在最常用的深度学习框架（PyTorch和TensorFlow）中复现基线方法，我们开展了一项基准研究，使用所提出的数据划分与评估指标进行统一评估，并公开发布以支持后续研究。所提出的数据划分方案与评估指标将推动全球范围内该数据集研究进展的跟踪，并促进后续部署中优化模型的选择。