We introduce LightIt, a method for explicit illumination control for image generation. Recent generative methods lack lighting control, which is crucial to numerous artistic aspects of image generation such as setting the overall mood or cinematic appearance. To overcome these limitations, we propose to condition the generation on shading and normal maps. We model the lighting with single bounce shading, which includes cast shadows. We first train a shading estimation module to generate a dataset of real-world images and shading pairs. Then, we train a control network using the estimated shading and normals as input. Our method demonstrates high-quality image generation and lighting control in numerous scenes. Additionally, we use our generated dataset to train an identity-preserving relighting model, conditioned on an image and a target shading. Our method is the first that enables the generation of images with controllable, consistent lighting and performs on par with specialized relighting state-of-the-art methods.

翻译：我们提出LightIt，一种用于图像生成的显式照明控制方法。当前的生成方法缺乏照明控制能力，而照明控制在图像生成的众多艺术层面（如设定整体氛围或电影感外观）中至关重要。为克服这些局限，我们提出以着色图和法线图作为生成条件。我们采用包含投射阴影的单次反射着色模型对光照进行建模。首先训练一个着色估计模块，用以生成包含真实图像与着色图配对的数据集；随后利用估计得到的着色图和法线图作为输入，训练一个控制网络。该方法在多种场景下实现了高质量的图像生成与照明控制。此外，我们利用生成的数据集训练了一个基于图像与目标着色图的条件化身份保持重光照模型。本方法首次实现了可控且一致照明的图像生成，其性能可与专业级重光照方法的最新成果相媲美。