Segmenting unseen objects from images is a critical perception skill that a robot needs to acquire. In robot manipulation, it can facilitate a robot to grasp and manipulate unseen objects. Mean shift clustering is a widely used method for image segmentation tasks. However, the traditional mean shift clustering algorithm is not differentiable, making it difficult to integrate it into an end-to-end neural network training framework. In this work, we propose the Mean Shift Mask Transformer (MSMFormer), a new transformer architecture that simulates the von Mises-Fisher (vMF) mean shift clustering algorithm, allowing for the joint training and inference of both the feature extractor and the clustering. Its central component is a hypersphere attention mechanism, which updates object queries on a hypersphere. To illustrate the effectiveness of our method, we apply MSMFormer to unseen object instance segmentation. Our experiments show that MSMFormer achieves competitive performance compared to state-of-the-art methods for unseen object instance segmentation. The project page, appendix, video, and code are available at https://irvlutd.github.io/MSMFormer

翻译：从图像中分割未见物体是机器人需要获取的关键感知技能。在机器人操作中，这可以帮助机器人抓取和操作未见物体。均值偏移聚类是图像分割任务中广泛使用的方法。然而，传统均值偏移聚类算法不可微分，难以整合到端到端神经网络训练框架中。本文提出均值偏移掩膜变换器（MSMFormer），一种新型变换器架构，模拟von Mises-Fisher（vMF）均值偏移聚类算法，实现特征提取器和聚类的联合训练与推理。其核心组件是超球面注意力机制，在超球面上更新物体查询。为验证方法有效性，我们将MSMFormer应用于未见物体实例分割。实验表明，MSMFormer在未见物体实例分割任务上取得了与最先进方法相当的性能。项目页面、附录、视频和代码见 https://irvlutd.github.io/MSMFormer