The emerging trend of AR/VR places great demands on 3D content. However, most existing software requires expertise and is difficult for novice users to use. In this paper, we aim to create sketch-based modeling tools for user-friendly 3D modeling. We introduce Reality3DSketch with a novel application of an immersive 3D modeling experience, in which a user can capture the surrounding scene using a monocular RGB camera and can draw a single sketch of an object in the real-time reconstructed 3D scene. A 3D object is generated and placed in the desired location, enabled by our novel neural network with the input of a single sketch. Our neural network can predict the pose of a drawing and can turn a single sketch into a 3D model with view and structural awareness, which addresses the challenge of sparse sketch input and view ambiguity. We conducted extensive experiments synthetic and real-world datasets and achieved state-of-the-art (SOTA) results in both sketch view estimation and 3D modeling performance. According to our user study, our method of performing 3D modeling in a scene is $>$5x faster than conventional methods. Users are also more satisfied with the generated 3D model than the results of existing methods.

翻译：增强现实/虚拟现实的兴起对三维内容提出了巨大需求。然而，现有软件大多需要专业知识，新手用户难以使用。本文致力于构建基于草图的三维建模工具，实现用户友好的建模体验。我们提出Reality3DSketch，创新性地实现沉浸式三维建模：用户可通过单目RGB相机捕捉周围场景，在实时重建的三维场景中绘制物体草图。通过我们提出的新型神经网络架构，系统仅需输入单幅草图即可生成三维物体并放置在指定位置。该神经网络能够预测绘制的姿态，并具有视角与结构感知能力地将单幅草图转化为三维模型，有效解决了草图输入稀疏与视角模糊性的挑战。我们在合成数据集与真实数据集上进行了广泛实验，在草图视角估计与三维建模性能两方面均达到最优水平。用户研究表明，我们的场景内三维建模方法比传统方法快5倍以上，且用户对生成三维模型的满意度优于现有方法。