Learning aligned multimodal embeddings from weakly paired, label-free corpora is challenging: pipelines often provide only pre-extracted features, clips contain multiple events, and spurious co-occurrences. We propose HSC-MAE (Hierarchical Semantic Correlation-Aware Masked Autoencoder), a dual-path teacher-student framework that enforces semantic consistency across three complementary levels of representation - from coarse to fine: (i) global-level canonical-geometry correlation via DCCA, which aligns audio and visual embeddings within a shared modality-invariant subspace; (ii) local-level neighborhood-semantics correlation via teacher-mined soft top-k affinities, which preserves multi-positive relational structure among semantically similar instances; and (iii) sample-level conditional-sufficiency correlation via masked autoencoding, which ensures individual embeddings retain discriminative semantic content under partial observation. Concretely, a student MAE path is trained with masked feature reconstruction and affinity-weighted soft top-k InfoNCE; an EMA teacher operating on unmasked inputs via the CCA path supplies stable canonical geometry and soft positives. Learnable multi-task weights reconcile competing objectives, and an optional distillation loss transfers teacher geometry into the student. Experiments on AVE and VEGAS demonstrate substantial mAP improvements over strong unsupervised baselines, validating that HSC-MAE yields robust and well-structured audio-visual representations.

翻译：从弱配对、无标签语料库中学习对齐的多模态嵌入具有挑战性：现有流程通常仅提供预提取特征，片段中包含多个事件，并存在伪共现现象。本文提出HSC-MAE（分级语义相关感知掩码自编码器），一种双路径教师-学生框架，在三个互补的表征层级上（从粗到细）强制实现语义一致性：（i）通过DCCA实现的全局层规范几何相关，在共享模态不变子空间中对齐音频与视觉嵌入；（ii）通过教师挖掘的软top-k亲和度实现的局部层邻域语义相关，保留语义相似实例间的多正例关系结构；（iii）通过掩码自编码实现的样本层条件充分相关，确保单个嵌入在部分观测条件下仍保留判别性语义内容。具体而言，学生MAE路径通过掩码特征重建与亲和度加权软top-k InfoNCE进行训练；通过CCA路径在未掩码输入上运行的EMA教师提供稳定的规范几何结构与软正例。可学习多任务权重调和竞争性目标，可选的蒸馏损失将教师几何结构迁移至学生。在AVE和VEGAS数据集上的实验表明，相较于强无监督基线获得显著mAP提升，验证了HSC-MAE能够生成鲁棒且结构良好的音视频表征。