Accents, as variations from standard pronunciation, pose significant challenges for speech recognition systems. Although joint automatic speech recognition (ASR) and accent recognition (AR) training has been proven effective in handling multi-accent scenarios, current multi-task ASR-AR approaches overlook the granularity differences between tasks. Fine-grained units capture pronunciation-related accent characteristics, while coarse-grained units are better for learning linguistic information. Moreover, an explicit interaction of two tasks can also provide complementary information and improve the performance of each other, but it is rarely used by existing approaches. In this paper, we propose a novel Decoupling and Interacting Multi-task Network (DIMNet) for joint speech and accent recognition, which is comprised of a connectionist temporal classification (CTC) branch, an AR branch, an ASR branch, and a bottom feature encoder. Specifically, AR and ASR are first decoupled by separated branches and two-granular modeling units to learn task-specific representations. The AR branch is from our previously proposed linguistic-acoustic bimodal AR model and the ASR branch is an encoder-decoder based Conformer model. Then, for the task interaction, the CTC branch provides aligned text for the AR task, while accent embeddings extracted from our AR model are incorporated into the ASR branch's encoder and decoder. Finally, during ASR inference, a cross-granular rescoring method is introduced to fuse the complementary information from the CTC and attention decoder after the decoupling. Our experiments on English and Chinese datasets demonstrate the effectiveness of the proposed model, which achieves 21.45%/28.53% AR accuracy relative improvement and 32.33%/14.55% ASR error rate relative reduction over a published standard baseline, respectively.

翻译：口音作为标准发音的变体，给语音识别系统带来了重大挑战。尽管联合自动语音识别与口音识别训练已被证明在多口音场景中有效，但当前的多任务ASR-AR方法忽略了任务间的粒度差异。细粒度单元能捕捉与发音相关的口音特征，而粗粒度单元更适合学习语言信息。此外，两个任务之间的显式交互可提供互补信息并提升彼此性能，但现有方法鲜有采用。本文提出一种新颖的解耦与交互多任务网络（DIMNet），用于联合语音与口音识别，该网络由连接时序分类分支、AR分支、ASR分支及底层特征编码器组成。具体而言，AR与ASR首先通过独立分支和双粒度建模单元解耦，以学习任务特定表征。AR分支源自我们先前提出的语言-声学双模态AR模型，ASR分支则为基于编码器-解码器的Conformer模型。随后，为实现任务交互，CTC分支为AR任务提供对齐文本，而从AR模型中提取的口音嵌入被融入ASR分支的编码器和解码器。最后，在ASR推理阶段，引入跨粒度重评分方法，融合解耦后CTC分支与注意力解码器的互补信息。我们在英语和中文数据集上的实验证明了所提模型的有效性：与已发布的基线标准相比，AR准确率相对提升21.45%/28.53%，ASR错误率相对降低32.33%/14.55%。