Diffusion models have found phenomenal success as expressive priors for solving inverse problems, but their extension beyond natural images to more structured scientific domains remains limited. Motivated by applications in materials science, we aim to reduce the number of measurements required from an expensive imaging modality of interest, by leveraging side information from an auxiliary modality that is much cheaper to obtain. To deal with the non-differentiable and black-box nature of the forward model, we propose a framework to train a multimodal diffusion model over the joint modalities, turning inverse problems with black-box forward models into simple linear inpainting problems. Numerically, we demonstrate the feasibility of training diffusion models over materials imagery data, and show that our approach achieves superior image reconstruction by leveraging the available side information, requiring significantly less amount of data from the expensive microscopy modality.

翻译：扩散模型作为解决逆问题的表达性先验已取得显著成功，但其应用范围主要局限于自然图像，在更具结构性的科学领域中仍较为有限。受材料科学应用的启发，本研究旨在通过利用辅助模态（其获取成本远低于目标模态）提供的附带信息，减少从昂贵的目标成像模态中所需的测量次数。针对前向模型的不可微性与黑箱特性，我们提出了一种在多模态联合空间上训练扩散模型的框架，从而将具有黑箱前向模型的逆问题转化为简单的线性修复问题。数值实验表明，在材料图像数据上训练扩散模型具有可行性，且我们的方法通过有效利用附带信息实现了更优的图像重建效果，显著减少了对昂贵显微成像模态的数据依赖。