This paper presents EdgeSAM, an accelerated variant of the Segment Anything Model (SAM), optimized for efficient execution on edge devices with minimal compromise in performance. Our approach involves distilling the original ViT-based SAM image encoder into a purely CNN-based architecture, better suited for edge devices. We carefully benchmark various distillation strategies and demonstrate that task-agnostic encoder distillation fails to capture the full knowledge embodied in SAM. To overcome this bottleneck, we include both the prompt encoder and mask decoder in the distillation process, with box and point prompts in the loop, so that the distilled model can accurately capture the intricate dynamics between user input and mask generation. To mitigate dataset bias issues stemming from point prompt distillation, we incorporate a lightweight module within the encoder. EdgeSAM achieves a 40-fold speed increase compared to the original SAM, and it also outperforms MobileSAM, being 14 times as fast when deployed on edge devices while enhancing the mIoUs on COCO and LVIS by 2.3 and 3.2 respectively. It is also the first SAM variant that can run at over 30 FPS on an iPhone 14. Code and models are available at https://github.com/chongzhou96/EdgeSAM.

翻译：本文提出EdgeSAM，一种加速版Segment Anything Model（SAM）变体，专为在边缘设备上高效执行而优化，且性能损失极小。我们的方法是将原始基于ViT的SAM图像编码器蒸馏为纯CNN架构，以更好地适配边缘设备。我们仔细对比了多种蒸馏策略，并表明任务无关的编码器蒸馏无法完整捕获SAM所蕴含的全部知识。为克服此瓶颈，我们将提示编码器和掩码解码器一并纳入蒸馏过程，并在循环中加入框提示与点提示，使蒸馏模型能准确捕捉用户输入与掩码生成之间的复杂动态关系。为缓解点提示蒸馏导致的数据集偏差问题，我们在编码器中融入轻量级模块。EdgeSAM相较原始SAM实现了40倍速度提升，同时在边缘设备部署时比MobileSAM快14倍，并在COCO和LVIS数据集上将mIoU分别提升2.3和3.2。此外，EdgeSAM是首个能在iPhone 14上以超过30 FPS运行的SAM变体。代码与模型已发布于https://github.com/chongzhou96/EdgeSAM。