Recently, radiance field rendering, such as 3D Gaussian Splatting (3DGS), has shown immense potential in VR content creation due to its high-quality rendering and efficient production process. However, existing physics-based interaction systems for 3DGS can only perform simple and non-realistic simulations or demand extensive user input for complex scenes, primarily due to the absence of scene understanding. In this paper, we propose LIVE-GS, a highly realistic interactive VR system powered by LLM. After object-aware GS reconstruction, we prompt GPT-4o to analyze the physical properties of objects in the scene, which are used to guide physical simulations consistent with real phenomena. We also design a GPT-assisted GS inpainting module to fill the unseen area covered by manipulative objects. To perform a precise segmentation of Gaussian kernels, we propose a feature-mask segmentation strategy. To enable rich interaction, we further propose a computationally efficient physical simulation framework through an PBD-based unified interpolation method, supporting various physical forms such as rigid body, soft body, and granular materials. Our experimental results show that with the help of LLM's understanding and enhancement of scenes, our VR system can support complex and realistic interactions without additional manual design and annotation.

翻译：近年来，辐射场渲染技术，如3D高斯泼溅（3DGS），因其高质量的渲染效果和高效的生产流程，在虚拟现实内容创作中展现出巨大潜力。然而，现有的基于物理的3DGS交互系统只能执行简单且非真实的模拟，或对复杂场景需要大量用户输入，这主要是由于缺乏场景理解能力。本文提出LIVE-GS，一个由大语言模型驱动的高度逼真的交互式虚拟现实系统。在完成物体感知的GS重建后，我们提示GPT-4o分析场景中物体的物理属性，这些属性被用于指导符合真实物理现象的模拟。我们还设计了一个GPT辅助的GS修复模块，以填充被操作物体遮挡的不可见区域。为了实现高斯核的精确分割，我们提出了一种特征掩码分割策略。为了支持丰富的交互，我们进一步提出了一种基于位置动力学统一插值方法的计算高效物理模拟框架，支持刚体、软体和颗粒材料等多种物理形态。实验结果表明，借助大语言模型对场景的理解与增强，我们的虚拟现实系统能够支持复杂且逼真的交互，而无需额外的人工设计与标注。