Training a high performance end-to-end speech (E2E) processing model requires an enormous amount of labeled speech data, especially in the era of data-centric artificial intelligence. However, labeled speech data are usually scarcer and more expensive for collection, compared to textual data. We propose Latent Synthesis (LaSyn), an efficient textual data utilization framework for E2E speech processing models. We train a latent synthesizer to convert textual data into an intermediate latent representation of a pre-trained speech model. These pseudo acoustic representations of textual data augment acoustic data for model training. We evaluate LaSyn on low-resource automatic speech recognition (ASR) and spoken language understanding (SLU) tasks. For ASR, LaSyn improves an E2E baseline trained on LibriSpeech train-clean-100, with relative word error rate reductions over 22.3% on different test sets. For SLU, LaSyn improves our E2E baseline by absolute 4.1% for intent classification accuracy and 3.8% for slot filling SLU-F1 on SLURP, and absolute 4.49% and 2.25% for exact match (EM) and EM-Tree accuracies on STOP respectively. With fewer parameters, the results of LaSyn are competitive to published state-of-the-art works. The results demonstrate the quality of the augmented training data.

翻译：训练高性能端到端语音处理模型需要大量带标注的语音数据，尤其是在以数据为中心的人工智能时代。然而，与文本数据相比，带标注的语音数据通常更为稀缺且收集成本更高。我们提出潜在合成（LaSyn），一种面向端到端语音处理模型的高效文本数据利用框架。我们训练一个潜在合成器，将文本数据转换为预训练语音模型的中间潜在表示。这些文本数据的伪声学表示可增强用于模型训练的声学数据。我们在低资源自动语音识别和口语理解任务上评估了LaSyn。在自动语音识别任务中，基于LibriSpeech train-clean-100数据集训练的端到端基线模型，LaSyn在不同测试集上的相对词错误率降低超过22.3%。在口语理解任务中，LaSyn在SLURP数据集上将意图分类准确率绝对提升4.1%，槽填充SLU-F1分数绝对提升3.8%；在STOP数据集上将精确匹配和精确匹配树准确率分别绝对提升4.49%和2.25%。尽管参数量更少，LaSyn的结果仍可与已发表的最新工作相媲美。这些结果证明了增强训练数据的质量。