Image restoration problems are typically ill-posed in the sense that each degraded image can be restored in infinitely many valid ways. To accommodate this, many works generate a diverse set of outputs by attempting to randomly sample from the posterior distribution of natural images given the degraded input. Here we argue that this strategy is commonly of limited practical value because of the heavy tail of the posterior distribution. Consider for example inpainting a missing region of the sky in an image. Since there is a high probability that the missing region contains no object but clouds, any set of samples from the posterior would be entirely dominated by (practically identical) completions of sky. However, arguably, presenting users with only one clear sky completion, along with several alternative solutions such as airships, birds, and balloons, would better outline the set of possibilities. In this paper, we initiate the study of meaningfully diverse image restoration. We explore several post-processing approaches that can be combined with any diverse image restoration method to yield semantically meaningful diversity. Moreover, we propose a practical approach for allowing diffusion based image restoration methods to generate meaningfully diverse outputs, while incurring only negligent computational overhead. We conduct extensive user studies to analyze the proposed techniques, and find the strategy of reducing similarity between outputs to be significantly favorable over posterior sampling. Code and examples are available at https://noa-cohen.github.io/MeaningfulDiversityInIR.

翻译：图像恢复问题通常是不适定的，即每个退化图像可以通过无数有效方式恢复。为解决这一问题，许多研究通过尝试从给定退化输入条件下自然图像的后验分布中随机采样，生成多样化的输出集合。本文论证了该策略通常实践价值有限，原因在于后验分布的厚尾特性。例如，考虑对图像中天空缺失区域进行修复时，由于缺失区域极大概率仅包含云层而非实体对象，后验分布的任何采样结果都将完全由（近乎相同的）天空补全结果主导。然而，向用户仅呈现一个清晰的天空补全结果，同时提供飞艇、鸟类、气球等多种替代方案，实际上能更好地勾勒可能性空间。本文首次系统研究有意义的多样化图像恢复问题。我们探索了多种后处理方法，这些方法可与任意多样化图像恢复方法结合，生成语义上有意义的多样性输出。此外，我们提出一种实用方案，使基于扩散模型的图像恢复方法能够生成有意义的多样化输出，且仅增加可忽略的计算开销。通过大规模用户研究分析所提技术，我们发现降低输出间相似度的策略显著优于后验采样方法。代码与示例见https://noa-cohen.github.io/MeaningfulDiversityInIR。