End-to-end speech translation models have become a new trend in the research due to their potential of reducing error propagation. However, these models still suffer from the challenge of data scarcity. How to effectively make use of unlabeled or other parallel corpora from machine translation is promising but still an open problem. In this paper, we propose Cross Speech-Text Network (XSTNet), an end-to-end model for speech-to-text translation. XSTNet takes both speech and text as input and outputs both transcription and translation text. The model benefits from its three key design aspects: a self supervising pre-trained sub-network as the audio encoder, a multi-task training objective to exploit additional parallel bilingual text, and a progressive training procedure. We evaluate the performance of XSTNet and baselines on the MuST-C En-De/Fr/Ru datasets. XSTNet achieves state-of-the-art results on all three language directions with an average BLEU of 27.8, outperforming the previous best method by 3.7 BLEU. The code and the models will be released to the public.

翻译：终端到终端语音翻译模型(XSTNet)由于具有减少错误传播的潜力,已成为研究的新趋势。然而,这些模型仍然受到数据稀缺的挑战。如何有效地利用机器翻译中未贴标签或其他平行的子公司很有希望,但仍然是一个尚未解决的问题。我们在此文件中提议跨语音-文字网络(XSTNet),一个语音-文字翻译的端到端模式。XSTNet将语音和文字作为输入和输出,同时作为文字和翻译文本。该模型从三个关键设计方面获益:作为音频编码器的自我监督的预先培训的子网络,一个利用额外平行双语文本的多任务培训目标,以及一个渐进式培训程序。我们评估了XSTNet的性能和 MuST-C En-De/Fr/Ru数据集的基线。 XSTNet在所有三种语言方向取得最新的结果,平均为27.8,比3.7 BLEU的先进方法要差。代码和模型将向公众公布。