Visual segmentation seeks to partition images, video frames, or point clouds into multiple segments or groups. This technique has numerous real-world applications, such as autonomous driving, image editing, robot sensing, and medical analysis. Over the past decade, deep learning-based methods have made remarkable strides in this area. Recently, transformers, a type of neural network based on self-attention originally designed for natural language processing, have considerably surpassed previous convolutional or recurrent approaches in various vision processing tasks. Specifically, vision transformers offer robust, unified, and even simpler solutions for various segmentation tasks. This survey provides a thorough overview of transformer-based visual segmentation, summarizing recent advancements. We first review the background, encompassing problem definitions, datasets, and prior convolutional methods. Next, we summarize a meta-architecture that unifies all recent transformer-based approaches. Based on this meta-architecture, we examine various method designs, including modifications to the meta-architecture and associated applications. We also present several closely related settings, including 3D point cloud segmentation, foundation model tuning, domain-aware segmentation, efficient segmentation, and medical segmentation. Additionally, we compile and re-evaluate the reviewed methods on several well-established datasets. Finally, we identify open challenges in this field and propose directions for future research. The project page can be found at https://github.com/lxtGH/Awesome-Segmentation-With-Transformer. We will also continually monitor developments in this rapidly evolving field.

翻译：视觉分割旨在将图像、视频帧或点云划分为多个片段或组。该技术在自动驾驶、图像编辑、机器人感知和医学分析等诸多实际应用中具有广泛用途。过去十年间，基于深度学习方法在该领域取得了显著进展。近年来，Transformer——一种最初为自然语言处理设计的基于自注意力机制的神经网络——已在多种视觉处理任务中大幅超越了先前的卷积或循环方法。具体而言，视觉Transformer为各类分割任务提供了鲁棒、统一甚至更简洁的解决方案。本综述全面概述了基于Transformer的视觉分割方法，总结了近期进展。我们首先回顾背景知识，涵盖问题定义、数据集以及先前的卷积方法。接着，我们归纳出一种统一所有近期基于Transformer方法的元架构。基于此元架构，我们审视了多种方法设计，包括对元架构的修改及其相关应用。我们还介绍了若干密切相关的设置，包括3D点云分割、基础模型微调、领域感知分割、高效分割和医学分割。此外，我们在多个公认数据集上对综述的方法进行了整理与重新评估。最后，我们指出了该领域存在的开放挑战并提出了未来研究方向。项目页面详见https://github.com/lxtGH/Awesome-Segmentation-With-Transformer。我们将持续关注这一快速发展领域的最新动态。