The Segment Anything Model (SAM) has demonstrated its effectiveness in segmenting any part of 2D RGB images. However, SAM exhibits a stronger emphasis on texture information while paying less attention to geometry information when segmenting RGB images. To address this limitation, we propose the Segment Any RGBD (SAD) model, which is specifically designed to extract geometry information directly from images. Inspired by the natural ability of humans to identify objects through the visualization of depth maps, SAD utilizes SAM to segment the rendered depth map, thus providing cues with enhanced geometry information and mitigating the issue of over-segmentation. We further include the open-vocabulary semantic segmentation in our framework, so that the 3D panoptic segmentation is fulfilled. The project is available on https://github.com/Jun-CEN/SegmentAnyRGBD.

翻译：分割任意模型（SAM）在二维RGB图像任意部分分割中已展现出有效性。然而，SAM在分割RGB图像时更侧重纹理信息，对几何信息的关注不足。为解决这一局限，我们提出分割任意RGBD（SAD）模型，该模型专为直接从图像中提取几何信息而设计。受人类通过深度图可视化识别物体这一天然能力的启发，SAD利用SAM对渲染后的深度图进行分割，从而提供增强几何信息的线索，并缓解过分割问题。我们进一步将开放词汇语义分割纳入框架中，进而实现三维全景分割。项目代码详见https://github.com/Jun-CEN/SegmentAnyRGBD。