Multimodal Large Language Models (MLLMs) have demonstrated an excellent understanding of images and 3D data. However, both modalities have shortcomings in holistically capturing the appearance and geometry of objects. Meanwhile, Neural Radiance Fields (NeRFs), which encode information within the weights of a simple Multi-Layer Perceptron (MLP), have emerged as an increasingly widespread modality that simultaneously encodes the geometry and photorealistic appearance of objects. This paper investigates the feasibility and effectiveness of ingesting NeRF into MLLM. We create LLaNA, the first general-purpose NeRF-language assistant capable of performing new tasks such as NeRF captioning and Q\&A. Notably, our method directly processes the weights of the NeRF's MLP to extract information about the represented objects without the need to render images or materialize 3D data structures. Moreover, we build a dataset of NeRFs with text annotations for various NeRF-language tasks with no human intervention. Based on this dataset, we develop a benchmark to evaluate the NeRF understanding capability of our method. Results show that processing NeRF weights performs favourably against extracting 2D or 3D representations from NeRFs.

翻译：多模态大语言模型（MLLMs）在理解图像和三维数据方面已展现出卓越能力。然而，这两种模态在整体捕捉物体的外观与几何形状方面均存在不足。与此同时，神经辐射场（NeRFs）作为一种日益普及的模态，通过将信息编码于简单的多层感知机（MLP）权重中，能够同时表征物体的几何结构与逼真外观。本文探讨了将NeRF融入MLLM的可行性与有效性。我们创建了首个通用型NeRF-语言助手LLaNA，该模型能够执行NeRF描述生成与问答等新型任务。值得注意的是，我们的方法直接处理NeRF中MLP的权重以提取所表征物体的信息，无需渲染图像或构建三维数据结构。此外，我们构建了带文本标注的NeRF数据集，用于各类NeRF-语言任务，且全程无需人工干预。基于该数据集，我们开发了评估方法对NeRF理解能力的基准测试。结果表明，与从NeRF中提取二维或三维表征相比，直接处理NeRF权重的方法表现更优。