We propose GaussianTalker, a novel framework for real-time generation of pose-controllable talking heads. It leverages the fast rendering capabilities of 3D Gaussian Splatting (3DGS) while addressing the challenges of directly controlling 3DGS with speech audio. GaussianTalker constructs a canonical 3DGS representation of the head and deforms it in sync with the audio. A key insight is to encode the 3D Gaussian attributes into a shared implicit feature representation, where it is merged with audio features to manipulate each Gaussian attribute. This design exploits the spatial-aware features and enforces interactions between neighboring points. The feature embeddings are then fed to a spatial-audio attention module, which predicts frame-wise offsets for the attributes of each Gaussian. It is more stable than previous concatenation or multiplication approaches for manipulating the numerous Gaussians and their intricate parameters. Experimental results showcase GaussianTalker's superiority in facial fidelity, lip synchronization accuracy, and rendering speed compared to previous methods. Specifically, GaussianTalker achieves a remarkable rendering speed up to 120 FPS, surpassing previous benchmarks. Our code is made available at https://github.com/KU-CVLAB/GaussianTalker/ .

翻译：我们提出高斯说话人（GaussianTalker），一种用于实时生成姿态可控说话人头部的全新框架。该方法利用三维高斯泼溅（3DGS）的快速渲染能力，同时解决直接通过语音音频控制3DGS的挑战。高斯说话人构建头部的典型3DGS表示，并使其与音频同步变形。关键创新在于将三维高斯属性编码为共享隐式特征表示，并与音频特征融合以操控每个高斯属性。该设计利用空间感知特征并强制相邻点间的交互。特征嵌入随后被输入空间-音频注意力模块，该模块为每个高斯的属性预测逐帧偏移量。相比先前通过拼接或乘法操控大量高斯及其复杂参数的方法，本方法更加稳定。实验结果表明，与先前方法相比，高斯说话人在面部保真度、唇部同步精度和渲染速度方面均具有优越性。具体而言，高斯说话人实现了高达120 FPS的显著渲染速度，超越了先前基准。我们的代码已开源至https://github.com/KU-CVLAB/GaussianTalker/ 。