There has been a recent explosion of computer vision models which perform many tasks and are composed of an image encoder (usually a ViT) and an autoregressive decoder (usually a Transformer). However, most of this work simply presents one system and its results, leaving many questions regarding design decisions and trade-offs of such systems unanswered. In this work, we aim to provide such answers. We take a close look at autoregressive decoders for multi-task learning in multimodal computer vision, including classification, captioning, visual question answering, and optical character recognition. Through extensive systematic experiments, we study the effects of task and data mixture, training and regularization hyperparameters, conditioning type and specificity, modality combination, and more. Importantly, we compare these to well-tuned single-task baselines to highlight the cost incurred by multi-tasking. A key finding is that a small decoder learned on top of a frozen pretrained encoder works surprisingly well. We call this setup locked-image tuning with decoder (LiT-decoder). It can be seen as teaching a decoder to interact with a pretrained vision model via natural language.

翻译：近年来，能够执行多种任务的计算机视觉模型层出不穷，这些模型通常由图像编码器（通常是ViT）和自回归解码器（通常是Transformer）组成。然而，现有研究大多仅展示了单一系统及其结果，而未充分解答关于此类系统设计决策与权衡的诸多问题。本研究旨在填补这一空白。我们深入探讨了多模态计算机视觉中面向多任务学习的自回归解码器，涵盖分类、图像描述、视觉问答及光学字符识别等任务。通过大规模系统性实验，我们分析了任务与数据混合方式、训练与正则化超参数、条件化类型与特异性、模态组合等因素的影响。特别地，我们将这些模型与精心调优的单任务基线进行对比，以揭示多任务处理带来的性能代价。核心发现是：在冻结的预训练编码器基础上学习一个小型解码器即可取得惊人效果。我们将这一框架称为"锁定图像微调解码器"（LiT-decoder），可将其理解为通过自然语言训练解码器与预训练视觉模型交互的过程。