Existing segmentation models based on multimodal large language models (MLLMs), such as LISA, often struggle with novel or emerging entities due to their inability to incorporate up-to-date knowledge. To address this challenge, we introduce the Novel Emerging Segmentation Task (NEST), which focuses on segmenting (i) novel entities that MLLMs fail to recognize due to their absence from training data, and (ii) emerging entities that exist within the model's knowledge but demand up-to-date external information for accurate recognition. To support the study of NEST, we construct a NEST benchmark using an automated pipeline that generates news-related data samples for comprehensive evaluation. Additionally, we propose ROSE: Retrieval-Oriented Segmentation Enhancement, a plug-and-play framework designed to augment any MLLM-based segmentation model. ROSE comprises four key components. First, an Internet Retrieval-Augmented Generation module is introduced to employ user-provided multimodal inputs to retrieve real-time web information. Then, a Textual Prompt Enhancer enriches the model with up-to-date information and rich background knowledge, improving the model's perception ability for emerging entities. Furthermore, a Visual Prompt Enhancer is proposed to compensate for MLLMs' lack of exposure to novel entities by leveraging internet-sourced images. To maintain efficiency, a WebSense module is introduced to intelligently decide when to invoke retrieval mechanisms based on user input. Experimental results demonstrate that ROSE significantly boosts performance on the NEST benchmark, outperforming a strong Gemini-2.0 Flash-based retrieval baseline by 19.2 in gIoU.

翻译：现有基于多模态大语言模型（MLLMs）的分割模型（如LISA）因无法整合最新知识，常难以处理新出现或新兴实体。为解决这一挑战，我们提出新兴分割任务（NEST），专注于分割两类实体：（i）因未出现在训练数据中而无法被MLLMs识别的新颖实体；（ii）虽存在于模型知识库但需依赖最新外部信息才能准确识别的新兴实体。为支持NEST研究，我们通过自动化流水线构建了基于新闻数据样本的NEST基准测试集。此外，我们提出ROSE（基于检索导向的分割增强框架），这是一种即插即用型架构，可增强任意基于MLLMs的分割模型。ROSE包含四个核心组件：首先引入互联网检索增强生成模块，利用用户提供的多模态输入实时获取网络信息；其次，文本提示增强器通过注入实时信息与丰富背景知识，提升模型对新兴实体的感知能力；第三，视觉提示增强器通过利用互联网图像源，弥补MLLMs对新颖实体认知的不足；最后，WebSense模块通过智能判断用户输入需求，决定何时激活检索机制以维持效率。实验结果表明，ROSE在NEST基准测试中性能显著提升，在gIoU指标上比强基线模型Gemini-2.0 Flash检索方法提高了19.2%。