We propose HILBERT (HIerarchical Long-sequence Balanced Embedding with Reciprocal contrastive Training), a cross-attentive multimodal framework for learning document-level audio-text representations from long, segmented sequences in low-resource data settings. HILBERT leverages frozen pre-trained speech and language encoders to extract segment-level features, which are aggregated via cross-modal attention and self-attentive pooling to form modality-specific document representations and a joint cross-attentive embedding. To align modalities while preserving modality-specific structure under severe audio-text dimensional imbalance, we introduce a reciprocal dual contrastive objective that simultaneously aligns audio-to-joint and text-to-joint representations, rather than directly contrasting audio and text alone. Two auxiliary regularizers further stabilize long-sequence fusion: a Centered Kernel Alignment (CKA) loss that preserves structural consistency between each modality and the joint embedding, and a mutual information balancing loss that prevents dominance of a single modality by equalizing information flow from audio and text into the joint space. For downstream prediction, HILBERT employs a Mixture-of-Experts (MoE) classifier over concatenated audio, text, and joint representations to accommodate heterogeneous label regimes. Extensive evaluation across multiple audio-text backbone combinations demonstrates that HILBERT learns semantically meaningful long-sequence representations and achieves superior performance on highly imbalanced multi-class settings.

翻译：我们提出HILBERT（基于互逆对比训练的分层长序列平衡嵌入）框架，这是一种用于在低资源数据场景下从长分段序列中学习文档级音频-文本表征的跨注意力多模态框架。HILBERT利用冻结的预训练语音和语言编码器提取片段级特征，通过跨模态注意力和自注意力池化聚合，形成模态特定文档表征与联合跨注意力嵌入。为在严重音频-文本维度失衡下对齐模态并保持模态特定结构，我们引入互逆双对比目标，该目标同步对齐音频-联合表征与文本-联合表征，而非直接对比音频与文本。两项辅助正则化项进一步稳定长序列融合：一是中心核对齐损失，用于保持各模态与联合嵌入之间的结构一致性；二是互信息平衡损失，通过均衡音频与文本流入联合空间的信息量，防止单一模态主导。在下游预测中，HILBERT采用专家混合分类器处理拼接后的音频、文本及联合表征，以适应异构标签体系。在多种音频-文本骨干组合上的广泛评估表明，HILBERT能学习具有语义意义的长序列表征，并在高度不平衡的多分类场景中取得卓越性能。