One of the main challenges of multimodal learning is the need to combine heterogeneous modalities (e.g., video, audio, text). For example, video and audio are obtained at much higher rates than text and are roughly aligned in time. They are often not synchronized with text, which comes as a global context, e.g., a title, or a description. Furthermore, video and audio inputs are of much larger volumes, and grow as the video length increases, which naturally requires more compute dedicated to these modalities and makes modeling of long-range dependencies harder. We here decouple the multimodal modeling, dividing it into separate, focused autoregressive models, processing the inputs according to the characteristics of the modalities. We propose a multimodal model, called Mirasol3B, consisting of an autoregressive component for the time-synchronized modalities (audio and video), and an autoregressive component for the context modalities which are not necessarily aligned in time but are still sequential. To address the long-sequences of the video-audio inputs, we propose to further partition the video and audio sequences in consecutive snippets and autoregressively process their representations. To that end, we propose a Combiner mechanism, which models the audio-video information jointly within a timeframe. The Combiner learns to extract audio and video features from raw spatio-temporal signals, and then learns to fuse these features producing compact but expressive representations per snippet. Our approach achieves the state-of-the-art on well established multimodal benchmarks, outperforming much larger models. It effectively addresses the high computational demand of media inputs by both learning compact representations, controlling the sequence length of the audio-video feature representations, and modeling their dependencies in time.

翻译：多模态学习的主要挑战之一是需整合异质模态（如视频、音频与文本）。例如，视频与音频采样率远高于文本且大致时间对齐，却常与作为全局上下文（如标题或描述）的文本不同步。此外，视频与音频输入数据量随时长增加而激增，自然需要更多计算资源处理这些模态，并使得长程依赖建模更为困难。本文解耦多模态建模，将其分解为独立的聚焦型自回归模型，根据模态特性分别处理输入。我们提出多模态模型Mirasol3B，包含一个用于时间同步模态（音频与视频）的自回归组件，以及一个用于非时间对齐但仍是序列的上下文模态的自回归组件。为应对视频-音频输入的长序列问题，我们进一步将视频与音频序列划分为连续片段，并对其表示进行自回归处理。为此，我们提出组合器（Combiner）机制，可在时间窗内联合建模音频-视频信息。该组合器从原始时空信号中学习提取音频与视频特征，进而融合这些特征，为每个片段生成紧凑且富有表达力的表示。本方法在权威多模态基准上达到最优性能，超越众多更大规模模型。通过学习紧凑表示、控制音频-视频特征表示序列长度并建模其时序依赖，有效解决了媒体输入的高计算需求问题。