Comparing spoken segments is a central operation to speech processing. Traditional approaches in this area have favored frame-level dynamic programming algorithms, such as dynamic time warping, because they require no supervision, but they are limited in performance and efficiency. As an alternative, acoustic word embeddings -- fixed-dimensional vector representations of variable-length spoken word segments -- have begun to be considered for such tasks as well. However, the current space of such discriminative embedding models, training approaches, and their application to real-world downstream tasks is limited. We start by considering ``single-view" training losses where the goal is to learn an acoustic word embedding model that separates same-word and different-word spoken segment pairs. Then, we consider ``multi-view" contrastive losses. In this setting, acoustic word embeddings are learned jointly with embeddings of character sequences to generate acoustically grounded embeddings of written words, or acoustically grounded word embeddings. In this thesis, we contribute new discriminative acoustic word embedding (AWE) and acoustically grounded word embedding (AGWE) approaches based on recurrent neural networks (RNNs). We improve model training in terms of both efficiency and performance. We take these developments beyond English to several low-resource languages and show that multilingual training improves performance when labeled data is limited. We apply our embedding models, both monolingual and multilingual, to the downstream tasks of query-by-example speech search and automatic speech recognition. Finally, we show how our embedding approaches compare with and complement more recent self-supervised speech models.

翻译：口语段落的比较是语音处理的核心操作。传统方法主要采用帧级动态规划算法（如动态时间规整），这些方法无需监督学习，但性能和效率受限。作为替代方案，声学词嵌入——将变长口语词段映射为固定维度向量表征——已开始被应用于此类任务。然而，当前此类判别式嵌入模型、训练方法及其在真实下游任务中的应用仍存在局限。我们首先探讨"单视角"训练损失函数，其目标在于学习能区分同词与异词口语段落的声学词嵌入模型。继而研究"多视角"对比损失函数。在此设定下，声学词嵌入与字符序列嵌入被联合学习，以生成具有声学基础的文字词嵌入（即声学基础词嵌入）。本论文基于循环神经网络提出了新型判别式声学词嵌入和声学基础词嵌入方法。我们在模型训练效率和性能方面实现了双重提升。我们将这些方法从英语拓展至多种低资源语言，并证明在多标签数据受限时，多语言训练能提升模型性能。我们将单语及多语言嵌入模型应用于示例查询语音搜索和自动语音识别等下游任务。最后，我们展示了所提嵌入方法与最新自监督语音模型的性能对比与互补关系。