The recent success of CLIP has demonstrated promising results in zero-shot semantic segmentation by transferring muiltimodal knowledge to pixel-level classification. However, leveraging pre-trained CLIP knowledge to closely align text embeddings with pixel embeddings still has limitations in existing approaches. To address this issue, we propose OTSeg, a novel multimodal attention mechanism aimed at enhancing the potential of multiple text prompts for matching associated pixel embeddings. We first propose Multi-Prompts Sinkhorn (MPS) based on the Optimal Transport (OT) algorithm, which leads multiple text prompts to selectively focus on various semantic features within image pixels. Moreover, inspired by the success of Sinkformers in unimodal settings, we introduce the extension of MPS, called Multi-Prompts Sinkhorn Attention (MPSA) , which effectively replaces cross-attention mechanisms within Transformer framework in multimodal settings. Through extensive experiments, we demonstrate that OTSeg achieves state-of-the-art (SOTA) performance with significant gains on Zero-Shot Semantic Segmentation (ZS3) tasks across three benchmark datasets.

翻译：CLIP模型的最新成功展示了通过将多模态知识迁移至像素级分类，在零样本语义分割任务中取得了令人瞩目的成果。然而，现有方法在利用预训练的CLIP知识使文本嵌入与像素嵌入紧密对齐方面仍存在局限。为解决这一问题，我们提出了OTSeg——一种新颖的多模态注意力机制，旨在增强多个文本提示匹配相关像素嵌入的潜力。我们首先基于最优传输算法提出了多提示Sinkhorn方法，该方法引导多个文本提示有选择地关注图像像素中的不同语义特征。此外，受Sinkformer在单模态场景中成功的启发，我们进一步扩展了MPS方法，提出了多提示Sinkhorn注意力机制，该机制在多模态场景中有效替代了Transformer框架内的交叉注意力机制。通过大量实验，我们在三个基准数据集上的零样本语义分割任务中证明，OTSeg取得了显著的性能提升，达到了最先进的水平。