We propose manvr3d, a novel VR-ready platform for interactive human-in-the-loop cell tracking. We utilize VR controllers and eye-tracking hardware to facilitate rapid ground truth generation and proofreading for deep learning-based cell tracking models. Life scientists reconstruct the developmental history of organisms on the cellular level by analyzing 3D time-lapse microscopy images acquired at high spatio-temporal resolution. The reconstruction of such cell lineage trees traditionally involves tracking individual cells through all recorded time points, manually annotating their positions, and then linking them over time to create complete trajectories. Deep learning-based algorithms accelerate this process, yet depend heavily on manually-annotated high-quality ground truth data and curation. Visual representation of the image data in this process still relies primarily on 2D renderings, which greatly limits spatial understanding and navigation. In this work, we bridge the gap between deep learning-based cell tracking software and 3D/VR visualization to create a human-in-the-loop cell tracking system. We lift the incremental annotation, training and proofreading loop of the deep learning model into the 3rd dimension and apply natural user interfaces like hand gestures and eye tracking to accelerate the cell tracking workflow for life scientists.

翻译：我们提出了 manvr3d，这是一个新颖的、支持 VR 的交互式人在回路细胞追踪平台。我们利用 VR 控制器和眼动追踪硬件，为基于深度学习的细胞追踪模型促进快速真值生成与校对。生命科学家通过分析以高时空分辨率获取的三维延时显微图像，在细胞层面重建生物体的发育历史。此类细胞谱系树的重建传统上涉及在所有记录的时间点追踪单个细胞、手动标注其位置，然后随时间将其连接以创建完整的轨迹。基于深度学习的算法加速了这一过程，但严重依赖于手动标注的高质量真值数据及其整理。在此过程中，图像数据的视觉呈现仍主要依赖于二维渲染，这极大地限制了空间理解和导航。在本工作中，我们弥合了基于深度学习的细胞追踪软件与三维/VR 可视化之间的差距，创建了一个人在回路的细胞追踪系统。我们将深度学习模型的增量标注、训练和校对循环提升至第三维度，并应用手势和眼动追踪等自然用户界面，以加速生命科学家的细胞追踪工作流程。