In speech translation, leveraging multimodal data to improve model performance and address limitations of individual modalities has shown significant effectiveness. In this paper, we harness the complementary strengths of speech and text, which are disparate modalities. We observe three levels of modality gap between them, denoted by Modal input representation, Modal semantic, and Modal hidden states. To tackle these gaps, we propose \textbf{F}use-\textbf{S}peech-\textbf{T}ext (\textbf{FST}), a cross-modal model which supports three distinct input modalities for translation: speech, text, and fused speech-text. We leverage multiple techniques for cross-modal alignment and conduct a comprehensive analysis to assess its impact on speech translation, machine translation, and fused speech-text translation. We evaluate FST on MuST-C, GigaST, and newstest benchmark. Experiments show that the proposed FST achieves an average 34.0 BLEU on MuST-C En$\rightarrow$De/Es/Fr (vs SOTA +1.1 BLEU). Further experiments demonstrate that FST does not degrade on MT task, as observed in prior works. Instead, it yields an average improvement of 3.2 BLEU over the pre-trained MT model.

翻译：在语音翻译中，利用多模态数据提升模型性能并弥补单一模态的局限性已展现出显著成效。本文利用语音与文本这两种异构模态的互补优势，揭示了二者之间存在的三个层级模态差距：模态输入表征、模态语义及模态隐状态。为解决上述差距，我们提出**融合语音文本模型（FST）**——一种支持三种不同输入模态（语音、文本及融合语音文本）进行翻译的跨模态模型。我们采用多种跨模态对齐技术，并开展全面分析评估其对语音翻译、机器翻译及融合语音文本翻译的影响。在MuST-C、GigaST及newstest基准上的实验表明，所提出的FST模型在MuST-C英→德/西/法任务上平均BLEU值达34.0（相较现有最优模型提升1.1个BLEU值）。进一步实验表明，与以往研究不同，FST在机器翻译任务上未出现性能下降，反而在预训练机器翻译模型基础上平均提升3.2个BLEU值。