Visual segmentation seeks to partition images, video frames, or point clouds into multiple segments or groups. This technique has numerous real-world applications, such as autonomous driving, image editing, robot sensing, and medical analysis. Over the past decade, deep learning-based methods have made remarkable strides in this area. Recently, transformers, a type of neural network based on self-attention originally designed for natural language processing, have considerably surpassed previous convolutional or recurrent approaches in various vision processing tasks. Specifically, vision transformers offer robust, unified, and even simpler solutions for various segmentation tasks. This survey provides a thorough overview of transformer-based visual segmentation, summarizing recent advancements. We first review the background, encompassing problem definitions, datasets, and prior convolutional methods. Next, we summarize a meta-architecture that unifies all recent transformer-based approaches. Based on this meta-architecture, we examine various method designs, including modifications to the meta-architecture and associated applications. We also present several closely related settings, including 3D point cloud segmentation, foundation model tuning, domain-aware segmentation, efficient segmentation, and medical segmentation. Additionally, we compile and re-evaluate the reviewed methods on several well-established datasets. Finally, we identify open challenges in this field and propose directions for future research. The project page can be found at https://github.com/lxtGH/Awesome-Segmenation-With-Transformer. We will also continually monitor developments in this rapidly evolving field.

翻译：视觉分割旨在将图像、视频帧或点云划分为多个片段或组。该技术具有众多实际应用，例如自动驾驶、图像编辑、机器人感知和医学分析。过去十年中，基于深度学习的算法在此领域取得了显著进展。近期，Transformer——一种最初为自然语言处理设计的基于自注意力的神经网络——在各种视觉处理任务中已大幅超越先前的卷积或循环方法。具体而言，视觉Transformer为各类分割任务提供了鲁棒、统一甚至更简洁的解决方案。本综述全面概述了基于Transformer的视觉分割，总结了最新进展。我们首先回顾背景，涵盖问题定义、数据集和先前的卷积方法。接着，我们总结了一种统一所有近期基于Transformer方法的元架构。基于此元架构，我们审视了各种方法设计，包括对元架构的修改及相关应用。我们还介绍了若干密切相关的设定，包括3D点云分割、基础模型微调、域感知分割、高效分割和医学分割。此外，我们在多个常用数据集上编译并重新评估了所综述的方法。最后，我们指出了该领域的开放性挑战，并提出了未来研究方向。项目页面可访问https://github.com/lxtGH/Awesome-Segmenation-With-Transformer。我们还将持续关注这一快速发展领域的最新动态。