We present multimodal neural posterior estimation (MultiNPE), a method to integrate heterogeneous data from different sources in simulation-based inference with neural networks. Inspired by advances in attention-based deep fusion learning, it empowers researchers to analyze data from different domains and infer the parameters of complex mathematical models with increased accuracy. We formulate different multimodal fusion approaches for MultiNPE (early, late, and hybrid) and evaluate their performance in three challenging numerical experiments. MultiNPE not only outperforms na\"ive baselines on a benchmark model, but also achieves superior inference on representative scientific models from neuroscience and cardiology. In addition, we systematically investigate the impact of partially missing data on the different fusion strategies. Across our different experiments, late and hybrid fusion techniques emerge as the methods of choice for practical applications of multimodal simulation-based inference.

翻译：我们提出了多模态神经后验估计（MultiNPE），一种在基于模拟的推断中通过神经网络整合来自不同来源异质数据的方法。受基于注意力的深度融合学习进展的启发，该方法使研究人员能够分析来自不同领域的数据，并以更高精度推断复杂数学模型的参数。我们为MultiNPE制定了不同的多模态融合方法（早期融合、晚期融合和混合融合），并在三个具有挑战性的数值实验中评估其性能。MultiNPE不仅在一个基准模型上优于朴素基线方法，而且在神经科学和心脏病学中具有代表性的科学模型上实现了更优的推断。此外，我们系统地研究了部分数据缺失对不同融合策略的影响。在各项实验中，晚期融合和混合融合技术成为多模态模拟推断实际应用中首选的方法。