Direct speech-to-speech translation (S2ST) with discrete units leverages recent progress in speech representation learning. Specifically, a sequence of discrete representations derived in a self-supervised manner are predicted from the model and passed to a vocoder for speech reconstruction, while still facing the following challenges: 1) Acoustic multimodality: the discrete units derived from speech with same content could be indeterministic due to the acoustic property (e.g., rhythm, pitch, and energy), which causes deterioration of translation accuracy; 2) high latency: current S2ST systems utilize autoregressive models which predict each unit conditioned on the sequence previously generated, failing to take full advantage of parallelism. In this work, we propose TranSpeech, a speech-to-speech translation model with bilateral perturbation. To alleviate the acoustic multimodal problem, we propose bilateral perturbation (BiP), which consists of the style normalization and information enhancement stages, to learn only the linguistic information from speech samples and generate more deterministic representations. With reduced multimodality, we step forward and become the first to establish a non-autoregressive S2ST technique, which repeatedly masks and predicts unit choices and produces high-accuracy results in just a few cycles. Experimental results on three language pairs demonstrate that BiP yields an improvement of 2.9 BLEU on average compared with a baseline textless S2ST model. Moreover, our parallel decoding shows a significant reduction of inference latency, enabling speedup up to 21.4x than autoregressive technique. Audio samples are available at \url{https://TranSpeech.github.io/}

翻译：直接语音到语音翻译（S2ST）结合离散单元技术，利用了语音表征学习的最新进展。具体而言，模型通过自监督方式预测离散表征序列，并将其传递给声码器进行语音重构，但仍面临以下挑战：1）声学多模态性：由于声学属性（如节奏、音高和能量），从相同内容的语音中导出的离散单元可能具有不确定性，导致翻译准确率下降；2）高延迟：当前S2ST系统采用自回归模型，需在已生成序列条件下逐单元预测，无法充分利用并行计算优势。本文提出TranSpeech——一种基于双边扰动的语音到语音翻译模型。为缓解声学多模态问题，我们提出双边扰动（BiP）方法，包含风格归一化和信息增强两个阶段，旨在仅从语音样本中学习语言信息并生成更具确定性的表征。通过降低多模态性，我们率先建立了非自回归S2ST技术，该技术通过重复掩码和预测单元选择，仅需少量循环即可生成高精度结果。在三个语言对上的实验结果表明，与基线无文本S2ST模型相比，BiP平均提升2.9个BLEU值。此外，我们的并行解码显著降低了推理延迟，相比自回归技术可实现高达21.4倍的加速。音频样本请访问\url{https://TranSpeech.github.io/}。