Advanced by transformer architecture, vision foundation models (VFMs) achieve remarkable progress in performance and generalization ability. Segment Anything Model (SAM) is one remarkable model that can achieve generalized segmentation. However, most VFMs cannot run in realtime, which makes it difficult to transfer them into several products. On the other hand, current real-time segmentation mainly has one purpose, such as semantic segmentation on the driving scene. We argue that diverse outputs are needed for real applications. Thus, this work explores a new real-time segmentation setting, named all-purpose segmentation in real-time, to transfer VFMs in real-time deployment. It contains three different tasks, including interactive segmentation, panoptic segmentation, and video segmentation. We aim to use one model to achieve the above tasks in real-time. We first benchmark several strong baselines. Then, we present Real-Time All Purpose SAM (RAP-SAM). It contains an efficient encoder and an efficient decoupled decoder to perform prompt-driven decoding. Moreover, we further explore different training strategies and tuning methods to boost co-training performance further. Our code and model are available at https://github.com/xushilin1/RAP-SAM/.

翻译：受Transformer架构推动，视觉基础模型（VFM）在性能和泛化能力方面取得了显著进展。Segment Anything Model（SAM）是能够实现通用分割的杰出模型之一。然而，大多数VFM无法实时运行，这使其难以迁移至产品化应用。另一方面，当前的实时分割主要服务于单一任务，例如驾驶场景下的语义分割。我们认为实际应用需要多样化输出，因此本研究探索了一种新的实时分割设置——实时全场景分割，旨在将VFM迁移至实时部署场景。该设置包含三项不同任务：交互式分割、全景分割和视频分割。我们致力于使用单一模型实时完成上述任务。首先，我们建立了多个强基线基准。随后，提出实时全场景分割模型（RAP-SAM），其包含高效编码器与高效解耦解码器，可执行提示驱动解码。此外，我们进一步探索了不同训练策略与微调方法以提升协同训练性能。代码与模型已开源至https://github.com/xushilin1/RAP-SAM/。