This paper presents a new framework for open-vocabulary semantic segmentation with the pre-trained vision-language model, named Side Adapter Network (SAN). Our approach models the semantic segmentation task as a region recognition problem. A side network is attached to a frozen CLIP model with two branches: one for predicting mask proposals, and the other for predicting attention bias which is applied in the CLIP model to recognize the class of masks. This decoupled design has the benefit CLIP in recognizing the class of mask proposals. Since the attached side network can reuse CLIP features, it can be very light. In addition, the entire network can be trained end-to-end, allowing the side network to be adapted to the frozen CLIP model, which makes the predicted mask proposals CLIP-aware. Our approach is fast, accurate, and only adds a few additional trainable parameters. We evaluate our approach on multiple semantic segmentation benchmarks. Our method significantly outperforms other counterparts, with up to 18 times fewer trainable parameters and 19 times faster inference speed. We hope our approach will serve as a solid baseline and help ease future research in open-vocabulary semantic segmentation. The code will be available at https://github.com/MendelXu/SAN.

翻译：本文提出了一种基于预训练视觉语言模型的新框架，用于开放词汇语义分割，命名为侧适配网络（SAN）。该方法将语义分割任务建模为区域识别问题。我们在冻结的CLIP模型上附加一个侧网络，该网络包含两个分支：一个用于预测掩膜提案，另一个用于预测注意力偏置，该偏置应用于CLIP模型以识别掩膜的类别。这种解耦设计有助于CLIP识别掩膜提案的类别。由于附加的侧网络可以复用CLIP特征，因此其结构非常轻量。此外，整个网络可端到端训练，使侧网络能自适应冻结的CLIP模型，从而生成对CLIP感知的掩膜提案。本方法速度快、精度高，仅需增加少量可训练参数。我们在多个语义分割基准上评估了该方法。与其他方法相比，我们的方法在训练参数减少高达18倍的同时，推理速度提升19倍。我们希望该工作能为开放词汇语义分割奠定坚实基础，并促进相关研究。代码将发布在https://github.com/MendelXu/SAN。