We present multimodal neural posterior estimation (MultiNPE), a method to integrate heterogeneous data from different sources in simulation-based inference with neural networks. Inspired by advances in deep fusion, it allows researchers to analyze data from different domains and infer the parameters of complex mathematical models with increased accuracy. We consider three fusion approaches for MultiNPE (early, late, hybrid) and evaluate their performance in three challenging experiments. MultiNPE not only outperforms single-source baselines on a reference task, but also achieves superior inference on scientific models from cognitive neuroscience and cardiology. We systematically investigate the impact of partially missing data on the different fusion strategies. Across our experiments, late and hybrid fusion techniques emerge as the methods of choice for practical applications of multimodal simulation-based inference.

翻译：我们提出了多模态神经后验估计（MultiNPE），一种在基于仿真的推断中利用神经网络整合来自不同来源的异构数据的方法。受深度融合技术进展的启发，该方法使研究人员能够分析来自不同领域的数据，并以更高的精度推断复杂数学模型的参数。我们为MultiNPE考虑了三种融合策略（早期融合、晚期融合、混合融合），并在三项具有挑战性的实验中评估了它们的性能。MultiNPE不仅在基准任务上超越了单源基线方法，还在认知神经科学与心脏学领域的科学模型中实现了更优的推断性能。我们系统研究了部分数据缺失对不同融合策略的影响。在所有实验中，晚期融合与混合融合技术成为多模态仿真推断实际应用中的优选方法。