We introduce StyleGaussian, a novel 3D style transfer technique that allows instant transfer of any image's style to a 3D scene at 10 frames per second (fps). Leveraging 3D Gaussian Splatting (3DGS), StyleGaussian achieves style transfer without compromising its real-time rendering ability and multi-view consistency. It achieves instant style transfer with three steps: embedding, transfer, and decoding. Initially, 2D VGG scene features are embedded into reconstructed 3D Gaussians. Next, the embedded features are transformed according to a reference style image. Finally, the transformed features are decoded into the stylized RGB. StyleGaussian has two novel designs. The first is an efficient feature rendering strategy that first renders low-dimensional features and then maps them into high-dimensional features while embedding VGG features. It cuts the memory consumption significantly and enables 3DGS to render the high-dimensional memory-intensive features. The second is a K-nearest-neighbor-based 3D CNN. Working as the decoder for the stylized features, it eliminates the 2D CNN operations that compromise strict multi-view consistency. Extensive experiments show that StyleGaussian achieves instant 3D stylization with superior stylization quality while preserving real-time rendering and strict multi-view consistency. Project page: https://kunhao-liu.github.io/StyleGaussian/

翻译：我们提出StyleGaussian，一种新颖的3D风格迁移技术，能够以每秒10帧的速度将任意图像风格即时迁移至3D场景。该技术基于3D高斯溅射（3DGS），在不牺牲实时渲染能力与多视角一致性的前提下实现风格迁移。其即时风格迁移通过三个步骤完成：嵌入、迁移与解码。首先，将2D VGG场景特征嵌入至重建的3D高斯体中；其次，根据参考风格图像对嵌入特征进行变换；最后，将变换后的特征解码为风格化RGB图像。StyleGaussian包含两项创新设计：其一，一种高效特征渲染策略——先在低维特征空间进行渲染，再将其映射至高维特征空间，同时完成VGG特征嵌入。该策略显著降低内存消耗，使3DGS能够处理高维大内存需求的特征。其二，一种基于K近邻的3D卷积神经网络，作为风格化特征解码器，消除了破坏严格多视角一致性的2D CNN操作。大量实验表明，StyleGaussian在保持实时渲染与严格多视角一致性的同时，实现了即时的3D风格化，且风格化质量优异。项目主页：https://kunhao-liu.github.io/StyleGaussian/