Recent works propose extending 3DGS with semantic feature vectors for simultaneous semantic segmentation and image rendering. However, these methods often treat the semantic and rendering branches separately, relying solely on 2D supervision while ignoring the 3D Gaussian geometry. Moreover, current adaptive strategies adapt the Gaussian set depending solely on rendering gradients, which can be insufficient in subtle or textureless regions. In this work, we propose a joint enhancement framework for 3D semantic Gaussian modeling that synergizes both semantic and rendering branches. Firstly, unlike conventional point cloud shape encoding, we introduce an anisotropic 3D Gaussian Chebyshev descriptor using the Laplace-Beltrami operator to capture fine-grained 3D shape details, thereby distinguishing objects with similar appearances and reducing reliance on potentially noisy 2D guidance. In addition, without relying solely on rendering gradient, we adaptively adjust Gaussian allocation and spherical harmonics with local semantic and shape signals, enhancing rendering efficiency through selective resource allocation. Finally, we employ a cross-scene knowledge transfer module to continuously update learned shape patterns, enabling faster convergence and robust representations without relearning shape information from scratch for each new scene. Experiments on multiple datasets demonstrate improvements in segmentation accuracy and rendering quality while maintaining high rendering frame rates.

翻译：近期研究提出通过扩展3DGS的语义特征向量实现同步语义分割与图像渲染。然而，这些方法往往将语义分支与渲染分支独立处理，仅依赖二维监督而忽略三维高斯几何特性。此外，现有自适应策略仅根据渲染梯度调整高斯集合，在细节模糊或纹理缺失区域可能失效。本研究提出一种联合增强框架，通过协同优化语义与渲染分支提升三维语义高斯建模性能。首先，区别于传统点云形状编码方法，我们引入基于拉普拉斯-贝尔特拉米算子的各向异性三维高斯切比雪夫描述符，以捕捉细粒度三维形状细节，从而有效区分外观相似物体并降低对可能存在噪声的二维引导的依赖。其次，通过结合局部语义与形状信号自适应调整高斯分布与球谐函数，而非仅依赖渲染梯度，实现基于选择性资源分配的渲染效率提升。最后，我们采用跨场景知识迁移模块持续更新学习到的形状模式，使模型无需在新场景中重新学习形状信息即可实现快速收敛与鲁棒表征。在多数据集上的实验表明，该方法在保持高渲染帧率的同时，显著提升了分割精度与渲染质量。