Image restoration problems are typically ill-posed in the sense that each degraded image can be restored in infinitely many valid ways. To accommodate this, many works generate a diverse set of outputs by attempting to randomly sample from the posterior distribution of natural images given the degraded input. Here we argue that this strategy is commonly of limited practical value because of the heavy tail of the posterior distribution. Consider for example inpainting a missing region of the sky in an image. Since there is a high probability that the missing region contains no object but clouds, any set of samples from the posterior would be entirely dominated by (practically identical) completions of sky. However, arguably, presenting users with only one clear sky completion, along with several alternative solutions such as airships, birds, and balloons, would better outline the set of possibilities. In this paper, we initiate the study of meaningfully diverse image restoration. We explore several post-processing approaches that can be combined with any diverse image restoration method to yield semantically meaningful diversity. Moreover, we propose a practical approach for allowing diffusion based image restoration methods to generate meaningfully diverse outputs, while incurring only negligent computational overhead. We conduct extensive user studies to analyze the proposed techniques, and find the strategy of reducing similarity between outputs to be significantly favorable over posterior sampling. Code and examples are available in https://noa-cohen.github.io/MeaningfulDiversityInIR

翻译：图像复原问题通常是不适定的，因为每张退化图像可以以无限种有效方式进行复原。为应对这一特性，许多工作通过尝试从给定退化输入的自然图像后验分布中随机采样来生成多样化的输出。本文论证了这一策略在实践中常具有有限价值，原因在于后验分布的重尾特性。以图像中天空缺失区域的修复为例：由于缺失区域极大概率不含物体而仅包含云层，后验采样的任何样本集都将完全由（几乎相同的）天空补全结果主导。然而，向用户仅展示一种清晰的天空补全结果，并辅以飞艇、鸟类和气球等若干替代方案，反而能更好地勾勒可能的复原空间。本文开创性地研究了具有语义多样性的图像复原方法。我们探索了多种可与任意多样性图像复原方法结合的后处理技术，以生成语义上有意义的多样化结果。此外，我们提出了一种实用方法，使基于扩散模型的图像复原方法能生成有意义的多样化输出，同时仅产生极少的计算开销。通过大规模用户研究分析所提出的技术，发现降低输出间相似性的策略明显优于后验采样。代码与示例详见 https://noa-cohen.github.io/MeaningfulDiversityInIR