We propose VQ-NeRF, a two-branch neural network model that incorporates Vector Quantization (VQ) to decompose and edit reflectance fields in 3D scenes. Conventional neural reflectance fields use only continuous representations to model 3D scenes, despite the fact that objects are typically composed of discrete materials in reality. This lack of discretization can result in noisy material decomposition and complicated material editing. To address these limitations, our model consists of a continuous branch and a discrete branch. The continuous branch follows the conventional pipeline to predict decomposed materials, while the discrete branch uses the VQ mechanism to quantize continuous materials into individual ones. By discretizing the materials, our model can reduce noise in the decomposition process and generate a segmentation map of discrete materials. Specific materials can be easily selected for further editing by clicking on the corresponding area of the segmentation outcomes. Additionally, we propose a dropout-based VQ codeword ranking strategy to predict the number of materials in a scene, which reduces redundancy in the material segmentation process. To improve usability, we also develop an interactive interface to further assist material editing. We evaluate our model on both computer-generated and real-world scenes, demonstrating its superior performance. To the best of our knowledge, our model is the first to enable discrete material editing in 3D scenes.

翻译：我们提出VQ-NeRF，一种融合向量量化（VQ）的双分支神经网络模型，用于分解和编辑三维场景中的反射场。传统神经反射场仅使用连续表示建模三维场景，而现实物体通常由离散材质构成。这种缺乏离散化的特性会导致材质分解噪声大且编辑复杂。为克服这些局限，我们的模型由连续分支和离散分支组成。连续分支沿用传统管线预测分解后的材质，而离散分支通过VQ机制将连续材质量化为离散材质。通过离散化材质，模型可减少分解过程的噪声并生成离散材质的语义分割图。用户可通过点击分割结果对应区域轻松选择特定材质进行编辑。此外，我们提出基于丢弃法的VQ码字排序策略，用于预测场景中的材质数量，从而降低材质分割过程的冗余性。为提升可用性，我们还开发了交互式界面以辅助材质编辑。在计算机生成场景与真实场景上的评估表明，该模型具有优越性能。据我们所知，本模型首次实现了三维场景的离散材质编辑。