In large language models (LLM), in-context learning (ICL) refers to performing new tasks by conditioning on small demonstrations provided in the input context. Recent advances in visual in-context learning (VICL) demonstrate promising capabilities for solving downstream tasks by unified vision-language models (VLMs). When the visual prompt and the target images originate from different visual tasks, can VLMs still enable VICL? In the paper, we propose a fully collaborative pipeline, i.e. T2T-VICL, for VLMs to investigate the potential of cross-task VICL. Fundamentally, we design a mechanism to generate and select text prompts that best implicitly describe the differences between two distinct low-level vision tasks, and construct the first cross-task VICL dataset. Building upon this, we propose a novel inference framework that combines perceptual score-based reasoning with traditional evaluation metrics to perform cross-task VICL. Our approach achieves top-tier results across twelve cross-task scenarios and second-tier performance in nine additional scenarios, unlocking the boundaries of cross-task VICL within VLMs.

翻译：在大语言模型（LLM）中，上下文学习（ICL）指的是通过输入上下文中提供的少量示例来执行新任务。视觉上下文学习（VICL）的最新进展表明，统一的视觉语言模型（VLM）在解决下游任务方面展现出有前景的能力。当视觉提示与目标图像源自不同的视觉任务时，VLM是否仍能实现VICL？本文提出了一种完全协同的流程，即T2T-VICL，用于探索VLM在跨任务VICL中的潜力。本质上，我们设计了一种机制来生成和选择最能隐式描述两个不同低级视觉任务之间差异的文本提示，并构建了首个跨任务VICL数据集。在此基础上，我们提出了一种新颖的推理框架，该框架将基于感知得分的推理与传统评估指标相结合，以执行跨任务VICL。我们的方法在十二个跨任务场景中取得了顶级结果，并在另外九个场景中获得了次优性能，从而解锁了VLM中跨任务VICL的边界。