Recent work has shown that diffusion models can serve as powerful neural rendering engines that can be leveraged for inserting virtual objects into images. However, unlike typical physics-based renderers, these neural rendering engines are limited by the lack of manual control over the lighting, which is often essential for improving or personalizing the desired image outcome. In this paper, we show that precise lighting control can be achieved for object relighting simply by providing a coarse shadow of the object. Indeed, we show that injecting only the desired shadow of the object into a pre-trained diffusion-based neural renderer enables it to accurately shade the object according to the desired light position, while properly harmonizing the object (and its shadow) within the target background image. Our method, SpotLight, leverages existing neural rendering approaches and achieves controllable relighting results with no additional training. We show that SpotLight achieves superior object compositing results, both quantitatively and perceptually, as confirmed by a user study, outperforming existing diffusion-based models specifically designed for relighting. We also demonstrate other applications, such as hand-scribbling shadows and full-image relighting, demonstrating its versatility.

翻译：近期研究表明，扩散模型可作为强大的神经渲染引擎，用于将虚拟物体插入图像中。然而，与典型的基于物理的渲染器不同，这些神经渲染引擎因缺乏对照明的手动控制而受限，而照明控制对于改善或个性化期望的图像效果通常至关重要。本文证明，仅通过提供物体的粗略阴影即可实现精确的物体重光照控制。具体而言，我们展示了将目标物体的期望阴影注入预训练的基于扩散的神经渲染器中，即可使其根据期望的光源位置准确地对物体进行着色，同时将物体（及其阴影）与目标背景图像进行恰当融合。我们的方法SpotLight利用现有的神经渲染方法，无需额外训练即可实现可控的重光照效果。通过用户研究证实，SpotLight在定量和感知层面均实现了优异的物体合成效果，其性能优于现有专门为重光照设计的基于扩散的模型。我们还展示了其他应用，如手绘阴影和全图像重光照，证明了该方法的通用性。