Conversational bilingual speech encompasses three types of utterances: two purely monolingual types and one intra-sententially code-switched type. In this work, we propose a general framework to jointly model the likelihoods of the monolingual and code-switch sub-tasks that comprise bilingual speech recognition. By defining the monolingual sub-tasks with label-to-frame synchronization, our joint modeling framework can be conditionally factorized such that the final bilingual output, which may or may not be code-switched, is obtained given only monolingual information. We show that this conditionally factorized joint framework can be modeled by an end-to-end differentiable neural network. We demonstrate the efficacy of our proposed model on bilingual Mandarin-English speech recognition across both monolingual and code-switched corpora.

翻译：连通双语语言包括三种发音:两种纯单一语言类型和一种流用密码转换类型。在这项工作中,我们提出了一个总体框架,以共同模拟单语和代码转换子任务的可能性,其中包括双语语言识别。通过用标签同步和框架同步来界定单语子任务,我们的联合建模框架可以有条件地设定要素,这样最后的双语产出,无论是否被代码转换,只能获得单语信息。我们表明,这一有条件的因子化联合框架可以通过端到端差异神经网络来建模。我们展示了我们提议的单语和代码转换的子公司双语汉语英语识别模式的有效性。