Although lip-to-speech synthesis (L2S) has achieved significant progress in recent years, current state-of-the-art methods typically rely on intermediate representations such as mel-spectrograms or discrete self-supervised learning (SSL) tokens. The potential of latent diffusion models (LDMs) in this task remains largely unexplored. In this paper, we introduce SLD-L2S, a novel L2S framework built upon a hierarchical subspace latent diffusion model. Our method aims to directly map visual lip movements to the continuous latent space of a pre-trained neural audio codec, thereby avoiding the information loss inherent in traditional intermediate representations. The core of our method is a hierarchical architecture that processes visual representations through multiple parallel subspaces, initiated by a subspace decomposition module. To efficiently enhance interactions within and between these subspaces, we design the diffusion convolution block (DiCB) as our network backbone. Furthermore, we employ a reparameterized flow matching technique to directly generate the target latent vectors. This enables a principled inclusion of speech language model (SLM) and semantic losses during training, moving beyond conventional flow matching objectives and improving synthesized speech quality. Our experiments show that SLD-L2S achieves state-of-the-art generation quality on multiple benchmark datasets, surpassing existing methods in both objective and subjective evaluations.

翻译：尽管唇语到语音合成（L2S）近年来取得了显著进展，但当前最先进的方法通常依赖于中间表示，如梅尔频谱图或离散自监督学习（SSL）标记。潜在扩散模型（LDMs）在此任务中的潜力在很大程度上仍未得到充分探索。本文提出了SLD-L2S，一种基于分层子空间潜在扩散模型的新型L2S框架。我们的方法旨在将视觉唇部运动直接映射到预训练神经音频编解码器的连续潜在空间，从而避免传统中间表示固有的信息损失。我们方法的核心是一个分层架构，通过多个并行子空间处理视觉表示，并由一个子空间分解模块启动。为了有效增强这些子空间内部及之间的交互，我们设计了扩散卷积块（DiCB）作为网络主干。此外，我们采用重参数化的流匹配技术来直接生成目标潜在向量。这使得在训练过程中能够原则性地纳入语音语言模型（SLM）和语义损失，超越了传统的流匹配目标，并提升了合成语音的质量。我们的实验表明，SLD-L2S在多个基准数据集上实现了最先进的生成质量，在客观和主观评估中均超越了现有方法。