3D generation has raised great attention in recent years. With the success of text-to-image diffusion models, the 2D-lifting technique becomes a promising route to controllable 3D generation. However, these methods tend to present inconsistent geometry, which is also known as the Janus problem. We observe that the problem is caused mainly by two aspects, i.e., viewpoint bias in 2D diffusion models and overfitting of the optimization objective. To address it, we propose a two-stage 2D-lifting framework, namely DreamControl, which optimizes coarse NeRF scenes as 3D self-prior and then generates fine-grained objects with control-based score distillation. Specifically, adaptive viewpoint sampling and boundary integrity metric are proposed to ensure the consistency of generated priors. The priors are then regarded as input conditions to maintain reasonable geometries, in which conditional LoRA and weighted score are further proposed to optimize detailed textures. DreamControl can generate high-quality 3D content in terms of both geometry consistency and texture fidelity. Moreover, our control-based optimization guidance is applicable to more downstream tasks, including user-guided generation and 3D animation. The project page is available at https://github.com/tyhuang0428/DreamControl.

翻译：近年来，3D生成技术受到广泛关注。随着文本到图像扩散模型的成功，2D提升技术成为可控3D生成的重要途径。然而，这类方法往往存在几何不一致性问题（即Janus问题）。我们观察到该问题主要由两方面因素导致：2D扩散模型的视角偏差和优化目标的过拟合。为此，我们提出一种两阶段2D提升框架DreamControl，该框架首先将粗粒度NeRF场景优化为3D自先验，然后通过基于控制的分数蒸馏生成细粒度物体。具体而言，我们提出自适应视角采样和边界完整性指标以确保生成先验的一致性。随后将这些先验作为输入条件以维持合理的几何结构，并进一步提出条件LoRA和加权分数来优化细节纹理。DreamControl在几何一致性和纹理保真度方面均能生成高质量3D内容。此外，我们的基于控制的优化指导可适用于更多下游任务，包括用户引导生成和3D动画。项目页面见https://github.com/tyhuang0428/DreamControl。