In-context learning$\unicode{x2013}$the ability to configure a model's behavior with different prompts$\unicode{x2013}$has revolutionized the field of natural language processing, alleviating the need for task-specific models and paving the way for generalist models capable of assisting with any query. Computer vision, in contrast, has largely stayed in the former regime: specialized decoders and finetuning protocols are generally required to perform dense tasks such as semantic segmentation and depth estimation. In this work we explore a simple mechanism for in-context learning of such scene understanding tasks: nearest neighbor retrieval from a prompt of annotated features. We propose a new pretraining protocol$\unicode{x2013}$leveraging attention within and across images$\unicode{x2013}$which yields representations particularly useful in this regime. The resulting Hummingbird model, suitably prompted, performs various scene understanding tasks without modification while approaching the performance of specialists that have been finetuned for each task. Moreover, Hummingbird can be configured to perform new tasks much more efficiently than finetuned models, raising the possibility of scene understanding in the interactive assistant regime.

翻译：情境学习——即通过不同提示配置模型行为的能力——已彻底改变了自然语言处理领域，消除了对特定任务模型的需求，并为能够协助任何查询的通用模型铺平了道路。相比之下，计算机视觉大多仍停留在前一种模式：执行语义分割和深度估计等密集任务通常需要专门的解码器和微调协议。在本研究中，我们探索了一种针对此类场景理解任务的情境学习简单机制：从标注特征的提示中进行最近邻检索。我们提出了一种新的预训练协议——利用图像内部及跨图像的注意力机制——该协议能产生在此模式下特别有用的表征。由此得到的Hummingbird模型，在适当提示下，无需修改即可执行各种场景理解任务，同时其性能接近为每个任务进行微调的专家模型。此外，与微调模型相比，Hummingbird能被更高效地配置以执行新任务，这为交互式助手模式下的场景理解提供了可能。