This paper reveals that large language models (LLMs), despite being trained solely on textual data, are surprisingly strong encoders for purely visual tasks in the absence of language. Even more intriguingly, this can be achieved by a simple yet previously overlooked strategy -- employing a frozen transformer block from pre-trained LLMs as a constituent encoder layer to directly process visual tokens. Our work pushes the boundaries of leveraging LLMs for computer vision tasks, significantly departing from conventional practices that typically necessitate a multi-modal vision-language setup with associated language prompts, inputs, or outputs. We demonstrate that our approach consistently enhances performance across a diverse range of tasks, encompassing pure 2D and 3D visual recognition tasks (e.g., image and point cloud classification), temporal modeling tasks (e.g., action recognition), non-semantic tasks (e.g., motion forecasting), and multi-modal tasks (e.g., 2D/3D visual question answering and image-text retrieval). Such improvements are a general phenomenon, applicable to various types of LLMs (e.g., LLaMA and OPT) and different LLM transformer blocks. We additionally propose the information filtering hypothesis to explain the effectiveness of pre-trained LLMs in visual encoding -- the pre-trained LLM transformer blocks discern informative visual tokens and further amplify their effect. This hypothesis is empirically supported by the observation that the feature activation, after training with LLM transformer blocks, exhibits a stronger focus on relevant regions. We hope that our work inspires new perspectives on utilizing LLMs and deepening our understanding of their underlying mechanisms. Code is available at https://github.com/ziqipang/LM4VisualEncoding.

翻译：本文揭示，大型语言模型（LLM）虽仅基于文本数据进行训练，但在缺乏语言辅助的纯视觉任务中却表现出惊人的编码能力。更令人感兴趣的是，这一能力可通过一种简单但此前被忽视的策略实现——将预训练LLM中的冻结Transformer块作为构成性编码层直接处理视觉标记。我们的工作拓展了利用LLM解决计算机视觉任务的边界，显著区别于传统做法（通常需要多模态视觉-语言框架以及相关的语言提示、输入或输出）。我们证明，该方法能够在多种任务中持续提升性能，涵盖纯2D与3D视觉识别任务（如图像、点云分类）、时序建模任务（如动作识别）、非语义任务（如运动预测）以及多模态任务（如2D/3D视觉问答与图像-文本检索）。这种改进具有普遍性，适用于多种类型的LLM（例如LLaMA与OPT）及其不同的Transformer块。此外，我们提出信息过滤假说以解释预训练LLM在视觉编码中的有效性——预训练LLM的Transformer块能识别信息性视觉标记并进一步放大其影响。该假说得到实验观察的支持：经LLM Transformer块训练后，特征激活对相关区域的关注度显著增强。我们期待这项工作能启发利用LLM的新视角，并深化对其内在机制的理解。代码详见https://github.com/ziqipang/LM4VisualEncoding。