Multimodal multiview learning seeks to integrate information from diverse sources to enhance task performance. Existing approaches often struggle with flexible view configurations, including arbitrary view combinations, numbers of views, and heterogeneous modalities. Focusing on the context of human activity recognition, we propose AliAd, a model that combines multiview contrastive learning with a mixture-of-experts module to support arbitrary view availability during both training and inference. Instead of trying to reconstruct missing views, an adjusted center contrastive loss is used for self-supervised representation learning and view alignment, mitigating the impact of missing views on multiview fusion. This loss formulation allows for the integration of view weights to account for view quality. Additionally, it reduces computational complexity from $O(V^2)$ to $O(V)$, where $V$ is the number of views. To address residual discrepancies not captured by contrastive learning, we employ a mixture-of-experts module with a specialized load balancing strategy, tasked with adapting to arbitrary view combinations. We highlight the geometric relationship among components in our model and how they combine well in the latent space. AliAd is validated on four datasets encompassing inertial and human pose modalities, with the number of views ranging from three to nine, demonstrating its performance and flexibility.

翻译：多模态多视角学习旨在整合来自不同来源的信息以提升任务性能。现有方法通常难以处理灵活的视角配置，包括任意视角组合、视角数量以及异构模态。聚焦于人体活动识别场景，我们提出AliAd模型，该模型结合了多视角对比学习与专家混合模块，以支持训练和推理期间任意视角的可用性。与尝试重建缺失视角不同，我们采用一种调整后的中心对比损失进行自监督表示学习与视角对齐，从而减轻缺失视角对多视角融合的影响。此损失公式允许集成视角权重以考量视角质量。此外，它将计算复杂度从$O(V^2)$降低至$O(V)$，其中$V$为视角数量。为处理对比学习未能捕捉的残余差异，我们采用一个带有专门负载均衡策略的专家混合模块，其任务是适应任意视角组合。我们强调了模型中各组件之间的几何关系，以及它们如何在潜在空间中良好结合。AliAd在涵盖惯性与人体姿态模态的四个数据集上进行了验证，视角数量从三到九不等，结果证明了其性能与灵活性。