We present CLUSTSEG, a general, transformer-based framework that tackles different image segmentation tasks (i.e., superpixel, semantic, instance, and panoptic) through a unified neural clustering scheme. Regarding queries as cluster centers, CLUSTSEG is innovative in two aspects:1) cluster centers are initialized in heterogeneous ways so as to pointedly address task-specific demands (e.g., instance- or category-level distinctiveness), yet without modifying the architecture; and 2) pixel-cluster assignment, formalized in a cross-attention fashion, is alternated with cluster center update, yet without learning additional parameters. These innovations closely link CLUSTSEG to EM clustering and make it a transparent and powerful framework that yields superior results across the above segmentation tasks.

翻译：我们提出了CLUSTSEG，这是一个通用的基于Transformer的框架，通过统一的神经聚类方案处理不同的图像分割任务（即超像素、语义、实例和全景分割）。将查询视为聚类中心，CLUSTSEG在两个方面具有创新性：1）聚类中心以异质方式初始化，以针对性地满足任务特定需求（例如，实例级或类别级区分性），而无需修改架构；以及2）以交叉注意力形式形式化的像素-聚类分配，与聚类中心更新交替进行，而无需学习额外参数。这些创新将CLUSTSEG与EM聚类紧密联系起来，使其成为一个透明且强大的框架，在以上分割任务中均取得了优越的结果。