This paper presents that the masked-modeling principle driving the success of large foundational vision models can be effectively applied to audio by making predictions in a latent space. We introduce Audio-based Joint-Embedding Predictive Architecture (A-JEPA), a simple extension method for self-supervised learning from the audio spectrum. Following the design of I-JPEA, our A-JEPA encodes visible audio spectrogram patches with a curriculum masking strategy via context encoder, and predicts the representations of regions sampled at well-designed locations. The target representations of those regions are extracted by the exponential moving average of context encoder, \emph{i.e.}, target encoder, on the whole spectrogram. We find it beneficial to transfer random block masking into time-frequency aware masking in a curriculum manner, considering the complexity of highly correlated in local time and frequency in audio spectrograms. To enhance contextual semantic understanding and robustness, we fine-tune the encoder with a regularized masking on target datasets, instead of input dropping or zero. Empirically, when built with Vision Transformers structure, we find A-JEPA to be highly scalable and sets new state-of-the-art performance on multiple audio and speech classification tasks, outperforming other recent models that use externally supervised pre-training.

翻译：本文提出，驱动大型基础视觉模型成功的掩码建模原理可通过在潜在空间中进行预测有效应用于音频。我们提出基于音频的联合嵌入预测架构（A-JEPA），这是一种从音频频谱进行自监督学习的简单扩展方法。遵循I-JEPA的设计，我们的A-JEPA通过上下文编码器采用课程掩码策略对可见音频频谱图块进行编码，并对精心设计位置采样区域的表征进行预测。这些区域的目标表征由上下文编码器的指数移动平均（即目标编码器）在整个频谱图上提取。考虑到音频频谱中局部时间与频率高度相关的复杂性，我们发现将随机块掩码以课程方式转换为时频感知掩码具有优势。为增强上下文语义理解与鲁棒性，我们在目标数据集上使用正则化掩码对编码器进行微调，而非采用输入丢弃或归零操作。实验表明，基于Vision Transformers结构构建时，A-JEPA展现出高度可扩展性，并在多项音频与语音分类任务上达到最新最优性能，超越了近期使用外部监督预训练的其他模型。