We introduce Grounded SAM, which uses Grounding DINO as an open-set object detector to combine with the segment anything model (SAM). This integration enables the detection and segmentation of any regions based on arbitrary text inputs and opens a door to connecting various vision models. As shown in Fig.1, a wide range of vision tasks can be achieved by using the versatile Grounded SAM pipeline. For example, an automatic annotation pipeline based solely on input images can be realized by incorporating models such as BLIP and Recognize Anything. Additionally, incorporating Stable-Diffusion allows for controllable image editing, while the integration of OSX facilitates promptable 3D human motion analysis. Grounded SAM also shows superior performance on open-vocabulary benchmarks, achieving 48.7 mean AP on SegInW (Segmentation in the wild) zero-shot benchmark with the combination of Grounding DINO-Base and SAM-Huge models.

翻译：我们提出Grounded SAM，利用Grounding DINO作为开放集目标检测器，与分割一切模型（SAM）相结合。这一集成使得基于任意文本输入的任意区域检测与分割成为可能，并为连接多种视觉模型打开了大门。如图1所示，通过使用多功能的Grounded SAM流水线，可以实现广泛的视觉任务。例如，仅基于输入图像的自动标注流水线可通过集成BLIP和Recognize Anything等模型来实现。此外，集成Stable-Diffusion可实现可控图像编辑，而集成OSX则促进了可提示的3D人体运动分析。Grounded SAM在开放词汇基准测试中也表现出卓越性能，在结合Grounding DINO-Base和SAM-Huge模型时，SegInW（野外分割）零样本基准测试的平均精度达到48.7。