Diffusion models have emerged as a powerful method of generative modeling across a range of fields, capable of producing stunning photo-realistic images from natural language descriptions. However, these models lack explicit control over the 3D structure in the generated images. Consequently, this hinders our ability to obtain detailed 3D annotations for the generated images or to craft instances with specific poses and distances. In this paper, we propose a simple yet effective method that incorporates 3D geometry control into diffusion models. Our method exploits ControlNet, which extends diffusion models by using visual prompts in addition to text prompts. We generate images of the 3D objects taken from 3D shape repositories (e.g., ShapeNet and Objaverse), render them from a variety of poses and viewing directions, compute the edge maps of the rendered images, and use these edge maps as visual prompts to generate realistic images. With explicit 3D geometry control, we can easily change the 3D structures of the objects in the generated images and obtain ground-truth 3D annotations automatically. This allows us to improve a wide range of vision tasks, e.g., classification and 3D pose estimation, in both in-distribution (ID) and out-of-distribution (OOD) settings. We demonstrate the effectiveness of our method through extensive experiments on ImageNet-100, ImageNet-R, PASCAL3D+, ObjectNet3D, and OOD-CV. The results show that our method significantly outperforms existing methods across multiple benchmarks, e.g., 3.8 percentage points on ImageNet-100 using DeiT-B and 3.5 percentage points on PASCAL3D+ & ObjectNet3D using NeMo.

翻译：扩散模型已成为跨领域生成建模的强大方法，能够根据自然语言描述生成令人惊叹的逼真图像。然而，这些模型缺乏对生成图像中三维结构的显式控制。这阻碍了我们在生成图像上获取详细的三维标注，或制作具有特定姿态和距离的实例。本文提出了一种简单而有效的方法，将三维几何控制融入扩散模型。该方法利用ControlNet，通过除文本提示外额外使用视觉提示来扩展扩散模型。我们从三维形状库（如ShapeNet和Objaverse）中生成三维物体的图像，从多种姿态和视角方向进行渲染，计算渲染图像的边缘图，并将这些边缘图作为视觉提示用于生成逼真图像。借助显式三维几何控制，我们能够轻松改变生成图像中物体的三维结构，并自动获取真实的三维标注。这使我们能够在分布内（ID）和分布外（OOD）场景下改进广泛的视觉任务，例如分类和三维姿态估计。通过ImageNet-100、ImageNet-R、PASCAL3D+、ObjectNet3D和OOD-CV上的大量实验，我们证明了该方法的有效性。结果表明，我们的方法在多个基准上显著优于现有方法，例如在ImageNet-100上使用DeiT-B时提升3.8个百分点，在PASCAL3D+和ObjectNet3D上使用NeMo时提升3.5个百分点。