Many high-level skills that are required for computer vision tasks, such as parsing questions, comparing and contrasting semantics, and writing descriptions, are also required in other domains such as natural language processing. In this paper, we ask whether it is possible to learn those skills from textual data and then transfer them to vision tasks without ever training on visual training data. Key to our approach is exploiting the joint embedding space of contrastively trained vision and language encoders. In practice, there can be systematic differences between embedding spaces for different modalities in contrastive models, and we analyze how these differences affect our approach and study strategies to mitigate this concern. We produce models using only text training data on four representative tasks: image captioning, visual entailment, visual question answering and visual news, and evaluate them on standard benchmarks using images. We find these models generally perform close to models trained on images, while surpassing prior work for captioning and visual entailment in this text only setting by over 9 points, and outperforming all prior work on visual news by over 30 points. We also showcase a variety of stylistic image captioning models that are trained using no image data and no human-curated language data, but instead using readily-available text data from books, the web, or language models.

翻译：计算机视觉任务所需的多项高级技能，如解析问题、语义比较对比、撰写描述等，同样在自然语言处理等其他领域中不可或缺。本文探讨是否可以从文本数据中习得这些技能，并在未经视觉训练数据训练的情况下将其迁移至视觉任务。我们方法的关键在于利用经过对比训练得到的视觉与语言编码器的联合嵌入空间。实际应用中，对比模型中不同模态的嵌入空间可能存在系统性差异，我们分析了这些差异如何影响该方法，并研究了缓解该问题的策略。我们仅使用文本训练数据，在四项代表性任务（图像描述、视觉蕴含、视觉问答及视觉新闻）上构建模型，并利用图像在标准基准上进行评估。结果表明，这些模型的性能总体接近基于图像训练的模型，在纯文本设置下的图像描述与视觉蕴含任务中超越先前工作超9个百分点，在视觉新闻任务上更是以超30个百分点的优势领先所有先前工作。此外，我们还展示了多种风格化图像描述模型——这些模型无需任何图像数据或人工整理的语言数据，仅需从书籍、网络或语言模型中获取现成文本数据即可完成训练。