Prompt quality plays a critical role in the performance of the Segment Anything Model (SAM), yet existing approaches often rely on heuristic or manually crafted prompts, limiting scalability and generalization. In this paper, we propose Point Prompt Defender, an adversarial reinforcement learning framework that adopts an attack-for-defense paradigm to automatically optimize point prompts. We construct a task-agnostic point prompt environment by representing image patches as nodes in a dual-space graph, where edges encode both physical and semantic distances. Within this environment, an attacker agent learns to activate a subset of prompts that maximally degrade SAM's segmentation performance, while a defender agent learns to suppress these disruptive prompts and restore accuracy. Both agents are trained using Deep Q-Networks with a reward signal based on segmentation quality variation. During inference, only the defender is deployed to refine arbitrary coarse prompt sets, enabling enhanced SAM segmentation performance across diverse tasks without retraining. Extensive experiments show that Point Prompt Defender effectively improves SAM's robustness and generalization, establishing a flexible, interpretable, and plug-and-play framework for prompt-based segmentation.

翻译：提示质量对Segment Anything Model（SAM）的性能至关重要，然而现有方法通常依赖启发式或人工设计的提示，限制了可扩展性与泛化能力。本文提出Point Prompt Defender——一种采用“攻击即防御”范式的对抗性强化学习框架，用于自动优化点提示。我们通过将图像块表示为双空间图中的节点来构建任务无关的点提示环境，其中边编码物理距离与语义距离。在此环境中，攻击者智能体学习激活能最大程度降低SAM分割性能的提示子集，而防御者智能体学习抑制这些干扰性提示以恢复分割精度。两个智能体均采用基于分割质量变化奖励信号的深度Q网络进行训练。在推理阶段，仅部署防御者智能体来优化任意粗粒度提示集，从而无需重新训练即可提升SAM在多样化任务中的分割性能。大量实验表明，Point Prompt Defender有效提升了SAM的鲁棒性与泛化能力，为基于提示的分割建立了一个灵活、可解释且即插即用的框架。