Recent work has shown that diffusion models can be used as powerful neural rendering engines that can be leveraged for inserting virtual objects into images. Unlike typical physics-based renderers, however, neural rendering engines are limited by the lack of manual control over the lighting setup, which is often essential for improving or personalizing the desired image outcome. In this paper, we show that precise lighting control can be achieved for object relighting simply by specifying the desired shadows of the object. Rather surprisingly, we show that injecting only the shadow of the object into a pre-trained diffusion-based neural renderer enables it to accurately shade the object according to the desired light position, while properly harmonizing the object (and its shadow) within the target background image. Our method, SpotLight, leverages existing neural rendering approaches and achieves controllable relighting results with no additional training. Specifically, we demonstrate its use with two neural renderers from the recent literature. We show that SpotLight achieves superior object compositing results, both quantitatively and perceptually, as confirmed by a user study, outperforming existing diffusion-based models specifically designed for relighting.

翻译：近期研究表明，扩散模型可作为强大的神经渲染引擎，用于将虚拟物体插入图像。然而，与典型的基于物理的渲染器不同，神经渲染引擎因缺乏对照明设置的手动控制而受限，而照明控制对于改善或个性化预期图像效果往往至关重要。本文证明，仅通过指定物体的预期阴影即可实现精确的物体重光照控制。令人惊讶的是，我们发现在预训练的基于扩散的神经渲染器中仅注入物体阴影，即可使其根据预期光源位置准确着色物体，同时将物体（及其阴影）与目标背景图像进行恰当融合。我们的方法SpotLight利用现有神经渲染方案，无需额外训练即可实现可控的重光照效果。具体而言，我们通过近期文献中的两种神经渲染器验证了其有效性。实验表明，SpotLight在物体合成效果上均优于现有专门用于重光照的扩散模型，这一结论通过用户研究在定量评估与感知评价中得到证实。