Accurate and robust multimodal speaker identification is essential for multimedia understanding and biometric authentication. However, real-world polyglot scenarios pose two key challenges: speaker-discriminative representations should generalize across languages, and the model should remain reliable when face information is unavailable. To address these challenges, we propose MRAF, a Missing-Token Prompted Reliability-Aware Fusion framework for polyglot speaker identification across complete-modality, missing-face, and cross-lingual scenarios. MRAF represents unavailable face inputs with a learnable missing token instead of fixed zero-valued features, providing a trainable representation of the missing visual state. This design reduces the distribution gap caused by missing inputs and allows subsequent reliability estimation and cross-modal fusion to operate within a unified token space. To adaptively integrate modalities with different reliability, MRAF further introduces a reliability-aware cross-attention fusion module, which estimates face and audio reliability scores, normalizes them into modality weights, and applies these weights to token representations before bidirectional cross-attention. In this way, the model can emphasize reliable modality cues while suppressing unreliable ones. During training, MRAF jointly optimizes multi-branch classification losses, audio-only knowledge distillation, and center loss to improve speaker discrimination and missing-modality robustness. Experiments on the official POLY-SIM 2026 test set demonstrate the effectiveness of the proposed framework. In the final evaluation, MRAF achieves 100% accuracy on P3 and P5, and obtains competitive results on the more challenging missing-face settings P4 and P6. The source code will be released at https://github.com/MSA-LMC/MRAF.
翻译:准确且鲁棒的多模态说话人识别对多媒体理解与生物特征认证至关重要。然而,真实多语种场景面临两大挑战:说话人判别性表征需跨语言泛化,且模型应在面部信息缺失时保持可靠性。为解决这些问题,我们提出MRAF——一种面向完整模态、面部缺失及跨语言场景的缺失标记提示可靠性感知融合框架,用于多语种说话人识别。MRAF采用可学习的缺失标记(而非固定零值特征)表示不可用的面部输入,从而提供对缺失视觉状态的可训练表征。该设计减少了缺失输入导致的分布差异,使后续可靠性估计与跨模态融合能在统一标记空间中进行。为自适应融合不同可靠性的模态,MRAF进一步引入可靠性感知交叉注意力融合模块:该模块估计面部与音频可靠性分数,将其归一化为模态权重,并在双向交叉注意力前将这些权重应用于标记表征。通过这种方式,模型可强化可靠模态线索同时抑制不可靠特征。训练阶段,MRAF联合优化多分支分类损失、仅音频知识蒸馏损失与中心损失,以提升说话人判别性与缺失模态鲁棒性。在官方POLY-SIM 2026测试集上的实验验证了所提框架的有效性。最终评估中,MRAF在P3与P5任务上实现100%准确率,并在更具挑战性的面部缺失场景P4与P6上取得具有竞争力的结果。源代码将发布于https://github.com/MSA-LMC/MRAF。