As data collection and simulation capabilities advance, multi-modal learning, the task of learning from multiple modalities and sources of data, is becoming an increasingly important area of research. Surrogate models that learn from data of multiple auxiliary modalities to support the modeling of a highly expensive quantity of interest have the potential to aid outer loop applications such as optimization, inverse problems, or sensitivity analyses when multi-modal data are available. We develop two multi-modal Bayesian neural network surrogate models and leverage conditionally conjugate distributions in the last layer to estimate model parameters using stochastic variational inference (SVI). We provide a method to perform this conjugate SVI estimation in the presence of partially missing observations. We demonstrate improved prediction accuracy and uncertainty quantification compared to uni-modal surrogate models for both scalar and time series data.

翻译：随着数据采集与模拟能力的提升，多模态学习——即从多种模态与数据来源中学习的任务——正成为日益重要的研究领域。利用多辅助模态数据构建代理模型以支持高成本目标量的建模，有望在多模态数据可用时辅助外环应用，如优化、反问题或敏感性分析。我们开发了两种多模态贝叶斯神经网络代理模型，并利用末层的条件共轭分布通过随机变分推断估计模型参数。针对部分观测缺失的情形，我们提出了一种执行共轭变分推断的方法。与单模态代理模型相比，我们证明了所提方法在标量数据和时间序列数据上的预测精度与不确定性量化均有提升。