Visual in-context learning (VICL) enables visual foundation models to handle multiple tasks by steering them with demonstrative prompts. The choice of such prompts largely influences VICL performance, standing out as a key challenge. Prior work has made substantial progress on prompt retrieval and reranking strategies, but mainly focuses on prompt images while overlooking labels. We reveal these approaches sometimes get visually similar but label-inconsistent prompts, which potentially degrade VICL performance. On the other hand, higher label consistency between query and prompts preferably indicates stronger VICL results. Motivated by these findings, we develop a framework named LaPR (Label-aware Prompt Retrieval), which highlights the role of labels in prompt selection. Our framework first designs an image-label joint representation for prompts to incorporate label cues explicitly. Besides, to handle unavailable query labels at test time, we introduce a mixture-of-expert mechanism to the dual encoders with query-adaptive routing. Each expert is expected to capture a specific label mode, while the router infers query-adaptive mixture weights and helps to learn label-aware representation. We carefully design alternative optimization for experts and router, with a VICL performance-guided contrastive loss and a label-guided contrastive loss, respectively. Extensive experiments show promising and consistent improvement of LaPR on in-context segmentation, detection, and colorization tasks. Moreover, LaPR generalizes well across feature extractors and cross-fold scenarios, suggesting the importance of label utilization in prompt retrieval for VICL. Code is available at https://github.com/luotc-why/CVPR26-LaPR.

翻译：视觉上下文学习（VICL）通过使用示例提示引导视觉基础模型处理多种任务。提示的选择显著影响VICL性能，成为关键挑战。先前工作已在提示检索和重排序策略上取得实质性进展，但主要关注提示图像而忽视标签。我们发现这些方法有时会检索到视觉相似但标签不一致的提示，这可能降低VICL性能。另一方面，查询与提示之间更高的标签一致性往往预示着更强的VICL结果。基于这些发现，我们开发了名为LaPR（标签感知提示检索）的框架，强调标签在提示选择中的作用。该框架首先设计图像-标签联合表示以明确融入标签线索。此外，为处理测试时查询标签不可用的情况，我们引入混合专家机制到双编码器中，并采用查询自适应路由。每个专家负责捕获特定标签模式，路由器推断查询自适应混合权重并辅助学习标签感知表示。我们精心设计了专家与路由器的交替优化，分别使用VICL性能引导对比损失和标签引导对比损失。大量实验表明，LaPR在上下文的图像分割、检测和着色任务上取得持续且显著的改进。此外，LaPR在特征提取器和跨折场景中均展现出良好泛化能力，揭示了标签利用在VICL提示检索中的重要性。代码见：https://github.com/luotc-why/CVPR26-LaPR。