We propose VQ-NeRF, a two-branch neural network model that incorporates Vector Quantization (VQ) to decompose and edit reflectance fields in 3D scenes. Conventional neural reflectance fields use only continuous representations to model 3D scenes, despite the fact that objects are typically composed of discrete materials in reality. This lack of discretization can result in noisy material decomposition and complicated material editing. To address these limitations, our model consists of a continuous branch and a discrete branch. The continuous branch follows the conventional pipeline to predict decomposed materials, while the discrete branch uses the VQ mechanism to quantize continuous materials into individual ones. By discretizing the materials, our model can reduce noise in the decomposition process and generate a segmentation map of discrete materials. Specific materials can be easily selected for further editing by clicking on the corresponding area of the segmentation outcomes. Additionally, we propose a dropout-based VQ codeword ranking strategy to predict the number of materials in a scene, which reduces redundancy in the material segmentation process. To improve usability, we also develop an interactive interface to further assist material editing. We evaluate our model on both computer-generated and real-world scenes, demonstrating its superior performance. To the best of our knowledge, our model is the first to enable discrete material editing in 3D scenes.

翻译：本文提出VQ-NeRF——一种融合向量量化（VQ）的双分支神经网络模型，用于三维场景中反射率场的分解与编辑。传统神经反射率场仅采用连续表示建模三维场景，而实际物体通常由离散材料构成。这种离散性缺失会导致材料分解噪声大且编辑复杂。针对上述局限，本模型包含连续分支与离散分支：连续分支遵循传统流程预测分解后的材料，离散分支则通过VQ机制将连续材料量化为离散单元。通过材料离散化，模型可降低分解过程中的噪声，并生成离散材料的分割图。用户仅需点击分割结果的对应区域，即可便捷选择特定材料进行后续编辑。此外，我们提出基于丢弃法的VQ码本排序策略以预测场景中的材料数量，从而减少材料分割过程的冗余性。为提升易用性，还开发了交互式界面辅助材料编辑。在计算机生成场景与真实场景上的实验表明，本模型具有优越性能。据我们所知，本模型首次实现了三维场景中的离散材料编辑。