Text-to-speech(TTS) has undergone remarkable improvements in performance, particularly with the advent of Denoising Diffusion Probabilistic Models (DDPMs). However, the perceived quality of audio depends not solely on its content, pitch, rhythm, and energy, but also on the physical environment. In this work, we propose ViT-TTS, the first visual TTS model with scalable diffusion transformers. ViT-TTS complement the phoneme sequence with the visual information to generate high-perceived audio, opening up new avenues for practical applications of AR and VR to allow a more immersive and realistic audio experience. To mitigate the data scarcity in learning visual acoustic information, we 1) introduce a self-supervised learning framework to enhance both the visual-text encoder and denoiser decoder; 2) leverage the diffusion transformer scalable in terms of parameters and capacity to learn visual scene information. Experimental results demonstrate that ViT-TTS achieves new state-of-the-art results, outperforming cascaded systems and other baselines regardless of the visibility of the scene. With low-resource data (1h, 2h, 5h), ViT-TTS achieves comparative results with rich-resource baselines.~\footnote{Audio samples are available at \url{https://ViT-TTS.github.io/.}}

翻译：文本到语音（TTS）技术随着去噪扩散概率模型（DDPMs）的出现取得了显著性能提升。然而，音频的感知质量不仅取决于其内容、音调、节奏和能量，还受物理环境影响。本文提出ViT-TTS，首个采用可扩展扩散Transformer的视觉TTS模型。ViT-TTS通过视觉信息补充音素序列以生成高感知质量音频，为AR和VR的实际应用开辟新路径，提供更具沉浸感和真实感的音频体验。针对视觉声学信息学习中的数据稀缺问题，我们：1）引入自监督学习框架以增强视觉文本编码器和去噪解码器；2）利用在参数量和容量上可扩展的扩散Transformer学习视觉场景信息。实验结果表明，无论场景可见性如何，ViT-TTS均取得了新的最优结果，优于级联系统及其他基线模型。在低资源数据（1小时、2小时、5小时）条件下，ViT-TTS仍能达到与高资源基线相当的性能。~\footnote{音频样本可在\url{https://ViT-TTS.github.io/}获取。}