End-to-end sign language translation (SLT) aims to convert sign language videos into spoken language texts directly without intermediate representations. It has been a challenging task due to the modality gap between sign videos and texts and the data scarcity of labeled data. To tackle these challenges, we propose a novel Cross-modality Data Augmentation (XmDA) framework to transfer the powerful gloss-to-text translation capabilities to end-to-end sign language translation (i.e. video-to-text) by exploiting pseudo gloss-text pairs from the sign gloss translation model. Specifically, XmDA consists of two key components, namely, cross-modality mix-up and cross-modality knowledge distillation. The former explicitly encourages the alignment between sign video features and gloss embeddings to bridge the modality gap. The latter utilizes the generation knowledge from gloss-to-text teacher models to guide the spoken language text generation. Experimental results on two widely used SLT datasets, i.e., PHOENIX-2014T and CSL-Daily, demonstrate that the proposed XmDA framework significantly and consistently outperforms the baseline models. Extensive analyses confirm our claim that XmDA enhances spoken language text generation by reducing the representation distance between videos and texts, as well as improving the processing of low-frequency words and long sentences.

翻译：端到端手语翻译（SLT）旨在直接通过手语视频生成口语文本，无需中间表示。由于手语视频与文本之间存在模态差异，且标注数据极度匮乏，该任务极具挑战性。为解决这些问题，我们提出了一种新颖的跨模态数据增强（XmDA）框架，通过利用手语词汇翻译模型生成的伪词汇-文本对，将强大的词汇到文本翻译能力迁移至端到端手语翻译（即视频到文本）任务。具体而言，XmDA包含两个关键组件：跨模态混合与跨模态知识蒸馏。前者通过显式促进手语视频特征与词汇嵌入的对齐来弥合模态差距，后者则利用词汇到文本教师模型的生成知识指导口语文本生成。在两个广泛使用的SLT数据集（PHOENIX-2014T和CSL-Daily）上的实验结果表明，所提出的XmDA框架始终显著优于基线模型。大量分析证实了我们的观点：XmDA通过缩小视频与文本间的表征距离、提升低频词与长句处理能力，增强了口语文本生成效果。