Accents, as variations from standard pronunciation, pose significant challenges for speech recognition systems. Although joint automatic speech recognition (ASR) and accent recognition (AR) training has been proven effective in handling multi-accent scenarios, current multi-task ASR-AR approaches overlook the granularity differences between tasks. Fine-grained units capture pronunciation-related accent characteristics, while coarse-grained units are better for learning linguistic information. Moreover, an explicit interaction of two tasks can also provide complementary information and improve the performance of each other, but it is rarely used by existing approaches. In this paper, we propose a novel Decoupling and Interacting Multi-task Network (DIMNet) for joint speech and accent recognition, which is comprised of a connectionist temporal classification (CTC) branch, an AR branch, an ASR branch, and a bottom feature encoder. Specifically, AR and ASR are first decoupled by separated branches and two-granular modeling units to learn task-specific representations. The AR branch is from our previously proposed linguistic-acoustic bimodal AR model and the ASR branch is an encoder-decoder based Conformer model. Then, for the task interaction, the CTC branch provides aligned text for the AR task, while accent embeddings extracted from our AR model are incorporated into the ASR branch's encoder and decoder. Finally, during ASR inference, a cross-granular rescoring method is introduced to fuse the complementary information from the CTC and attention decoder after the decoupling. Our experiments on English and Chinese datasets demonstrate the effectiveness of the proposed model, which achieves 21.45%/28.53% AR accuracy relative improvement and 32.33%/14.55% ASR error rate relative reduction over a published standard baseline, respectively.

翻译：口音作为标准发音的变异，给语音识别系统带来了显著挑战。尽管联合自动语音识别（ASR）与口音识别（AR）训练已被证明在多口音场景中有效，但当前的多任务ASR-AR方法忽略了任务间的粒度差异。细粒度单元能够捕捉与发音相关的口音特征，而粗粒度单元更适合学习语言信息。此外，两个任务的显式交互可提供互补信息并相互提升性能，但现有方法极少采用这一策略。本文提出一种新颖的解耦与交互多任务网络（DIMNet）用于联合语音与口音识别，该网络由连接主义时序分类（CTC）分支、AR分支、ASR分支以及底层特征编码器组成。具体而言，AR与ASR首先通过独立分支和双粒度建模单元进行解耦，以学习任务特定表征。AR分支基于我们先前提出的语言学-声学双模态AR模型，ASR分支则为基于编码器-解码器的Conformer模型。在任务交互方面，CTC分支为AR任务提供对齐文本，而从AR模型中提取的口音嵌入被整合到ASR分支的编码器和解码器中。最终，在ASR推理阶段，引入跨粒度重新评分方法以融合解耦后CTC与注意力解码器的互补信息。我们在英语和中文数据集上的实验证明了所提模型的有效性：与已发布的标准化基线相比，AR准确率相对提升21.45%/28.53%，ASR错误率相对降低32.33%/14.55%。