In this paper, we explore the zero-shot capability of the Segment Anything Model (SAM) for food image segmentation. To address the lack of class-specific information in SAM-generated masks, we propose a novel framework, called FoodSAM. This innovative approach integrates the coarse semantic mask with SAM-generated masks to enhance semantic segmentation quality. Besides, we recognize that the ingredients in food can be supposed as independent individuals, which motivated us to perform instance segmentation on food images. Furthermore, FoodSAM extends its zero-shot capability to encompass panoptic segmentation by incorporating an object detector, which renders FoodSAM to effectively capture non-food object information. Drawing inspiration from the recent success of promptable segmentation, we also extend FoodSAM to promptable segmentation, supporting various prompt variants. Consequently, FoodSAM emerges as an all-encompassing solution capable of segmenting food items at multiple levels of granularity. Remarkably, this pioneering framework stands as the first-ever work to achieve instance, panoptic, and promptable segmentation on food images. Extensive experiments demonstrate the feasibility and impressing performance of FoodSAM, validating SAM's potential as a prominent and influential tool within the domain of food image segmentation. We release our code at https://github.com/jamesjg/FoodSAM.

翻译：本文探索了分割一切模型（SAM）在食品图像分割中的零样本能力。针对SAM生成掩码缺乏类别特定信息的问题，我们提出了一种名为FoodSAM的创新框架。该方法将粗粒度语义掩码与SAM生成的掩码相结合，以提升语义分割质量。此外，我们认识到食品中的食材可被视为独立个体，这启发我们对食品图像进行实例分割。进一步地，FoodSAM通过集成目标检测器扩展其零样本能力以涵盖全景分割，从而有效捕捉非食品物体信息。受近期可提示分割成功经验的启发，我们还扩展了FoodSAM以支持多种提示变体的可提示分割。由此，FoodSAM成为能够多粒度分割食品物品的全方位解决方案。值得注意的是，该开创性框架首次实现了食品图像的实例分割、全景分割与可提示分割。大量实验验证了FoodSAM的可行性与卓越性能，证实了SAM作为食品图像分割领域重要影响力工具的潜力。我们在https://github.com/jamesjg/FoodSAM 公开发布了代码。