PSALM is a powerful extension of the Large Multi-modal Model (LMM) to address the segmentation task challenges. To overcome the limitation of the LMM being limited to textual output, PSALM incorporates a mask decoder and a well-designed input schema to handle a variety of segmentation tasks. This schema includes images, task instructions, conditional prompts, and mask tokens, which enable the model to generate and classify segmentation masks effectively. The flexible design of PSALM supports joint training across multiple datasets and tasks, leading to improved performance and task generalization. PSALM achieves superior results on several benchmarks, such as RefCOCO/RefCOCO+/RefCOCOg, COCO Panoptic Segmentation, and COCO-Interactive, and further exhibits zero-shot capabilities on unseen tasks, such as open-vocabulary segmentation, generalized referring expression segmentation and video object segmentation, making a significant step towards a GPT moment in computer vision. Through extensive experiments, PSALM demonstrates its potential to transform the domain of image segmentation, leveraging the robust visual understanding capabilities of LMMs as seen in natural language processing. Code and models are available at https://github.com/zamling/PSALM.

翻译：PSALM是大型多模态模型（LMM）的强大扩展，旨在应对分割任务的挑战。为克服LMM仅能输出文本的限制，PSALM引入了一个掩码解码器以及精心设计的输入模式，以处理多样化的分割任务。该模式包含图像、任务指令、条件提示和掩码标记，使得模型能够有效生成并分类分割掩码。PSALM的灵活设计支持跨多个数据集和任务的联合训练，从而提升性能与任务泛化能力。PSALM在多个基准测试中取得了优异结果，如RefCOCO/RefCOCO+/RefCOCOg、COCO全景分割和COCO交互式分割，并在未见任务上展现出零样本能力，如开放词汇分割、广义指代表达分割和视频目标分割，向计算机视觉领域的“GPT时刻”迈出了重要一步。通过大量实验，PSALM展示了其变革图像分割领域的潜力，借鉴了LMM在自然语言处理中展现的鲁棒视觉理解能力。代码和模型已开源至 https://github.com/zamling/PSALM。