We propose VQ-NeRF, a two-branch neural network model that incorporates Vector Quantization (VQ) to decompose and edit reflectance fields in 3D scenes. Conventional neural reflectance fields use only continuous representations to model 3D scenes, despite the fact that objects are typically composed of discrete materials in reality. This lack of discretization can result in noisy material decomposition and complicated material editing. To address these limitations, our model consists of a continuous branch and a discrete branch. The continuous branch follows the conventional pipeline to predict decomposed materials, while the discrete branch uses the VQ mechanism to quantize continuous materials into individual ones. By discretizing the materials, our model can reduce noise in the decomposition process and generate a segmentation map of discrete materials. Specific materials can be easily selected for further editing by clicking on the corresponding area of the segmentation outcomes. Additionally, we propose a dropout-based VQ codeword ranking strategy to predict the number of materials in a scene, which reduces redundancy in the material segmentation process. To improve usability, we also develop an interactive interface to further assist material editing. We evaluate our model on both computer-generated and real-world scenes, demonstrating its superior performance. To the best of our knowledge, our model is the first to enable discrete material editing in 3D scenes.

翻译：我们提出VQ-NeRF，一种结合向量量化（Vector Quantization, VQ）的双分支神经网络模型，用于分解和编辑三维场景中的反射场。传统神经反射场仅使用连续表示来建模三维场景，而实际物体通常由离散材质构成，这种离散性的缺失会导致材质分解存在噪声且编辑复杂。为解决这些局限，我们的模型包含连续分支和离散分支：连续分支遵循传统流程预测分解后的材质，离散分支则利用VQ机制将连续材质量化为独立离散单元。通过离散化材质，模型可降低分解过程中的噪声，并生成离散材质的语义分割图。用户仅需点击分割结果对应区域即可便捷选择特定材质进行编辑。此外，我们提出基于丢弃机制（dropout）的VQ码字排序策略以自动预测场景中的材质数量，减少材质分割中的冗余性。为提升可用性，我们还开发了交互界面辅助材质编辑。在计算机生成场景与真实场景上的评估表明，该模型具有优越性能。据我们所知，本模型首次实现了三维场景中的离散材质编辑。