Vision transformers (ViTs) have achieved impressive results on various computer vision tasks in the last several years. In this work, we study the capability of frozen ViTs, pretrained only on visual data, to generalize to audio-visual data without finetuning any of its original parameters. To do so, we propose a latent audio-visual hybrid (LAVISH) adapter that adapts pretrained ViTs to audio-visual tasks by injecting a small number of trainable parameters into every layer of a frozen ViT. To efficiently fuse visual and audio cues, our LAVISH adapter uses a small set of latent tokens, which form an attention bottleneck, thus, eliminating the quadratic cost of standard cross-attention. Compared to the existing modality-specific audio-visual methods, our approach achieves competitive or even better performance on various audio-visual tasks while using fewer tunable parameters and without relying on costly audio pretraining or external audio encoders. Our code is available at https://genjib.github.io/project_page/LAVISH/

翻译：视觉Transformer（Vision Transformers, ViTs）在过去几年中已在多项计算机视觉任务中取得了显著成果。本研究探讨了仅基于视觉数据预训练的冻结ViT（不微调任何原始参数）在音视频数据上的泛化能力。为此，我们提出了一种潜在音视频混合（Latent Audio-Visual Hybrid, LAVISH）适配器，通过向冻结ViT的每一层注入少量可训练参数，使其适应音视频任务。为高效融合视觉与音频线索，我们的LAVISH适配器使用一小部分潜在标记（latent tokens）形成注意力瓶颈，从而消除了标准交叉注意力中的二次计算开销。与现有模态特定的音视频方法相比，我们的方法在多项音视频任务中实现了具有竞争力甚至更优的性能，同时使用了更少的可调参数，且无需依赖昂贵的音频预训练或外部音频编码器。我们的代码已在https://genjib.github.io/project_page/LAVISH/ 公开。