Text-to-speech(TTS) has undergone remarkable improvements in performance, particularly with the advent of Denoising Diffusion Probabilistic Models (DDPMs). However, the perceived quality of audio depends not solely on its content, pitch, rhythm, and energy, but also on the physical environment. In this work, we propose ViT-TTS, the first visual TTS model with scalable diffusion transformers. ViT-TTS complement the phoneme sequence with the visual information to generate high-perceived audio, opening up new avenues for practical applications of AR and VR to allow a more immersive and realistic audio experience. To mitigate the data scarcity in learning visual acoustic information, we 1) introduce a self-supervised learning framework to enhance both the visual-text encoder and denoiser decoder; 2) leverage the diffusion transformer scalable in terms of parameters and capacity to learn visual scene information. Experimental results demonstrate that ViT-TTS achieves new state-of-the-art results, outperforming cascaded systems and other baselines regardless of the visibility of the scene. With low-resource data (1h, 2h, 5h), ViT-TTS achieves comparative results with rich-resource baselines.~\footnote{Audio samples are available at \url{https://ViT-TTS.github.io/.}}

翻译：文本转语音（TTS）技术随着去噪扩散概率模型（DDPMs）的引入取得了显著性能提升。然而，音频的感知质量不仅取决于其内容、音高、节奏和能量，还受物理环境影响。本文提出ViT-TTS，这是首个结合可扩展扩散Transformer的视觉TTS模型。ViT-TTS利用视觉信息补充音素序列，生成高感知质量的音频，为AR和VR的实际应用开辟新途径，提供更具沉浸感和真实感的音频体验。为解决视觉声学信息学习中的数据稀缺问题，我们：（1）引入自监督学习框架，增强视觉文本编码器和去噪解码器；（2）利用参数和容量可扩展的扩散Transformer学习视觉场景信息。实验结果表明，无论场景是否可见，ViT-TTS均达到新的最优性能，超越级联系统及其他基线模型。在低资源数据（1小时、2小时、5小时）条件下，ViT-TTS可与高资源基线取得相当结果。~\footnote{音频样本见\url{https://ViT-TTS.github.io/}}