Mapping speech tokens to the same feature space as text tokens has become the paradigm for the integration of speech modality into decoder-only large language models (LLMs). An alternative approach is to use an encoder-decoder architecture that incorporates speech features through cross-attention. This approach, however, has received less attention in the literature. In this work, we connect the Whisper encoder with ChatGLM3 and provide in-depth comparisons of these two approaches using Chinese automatic speech recognition (ASR) and name entity recognition (NER) tasks. We evaluate them not only by conventional metrics like the F1 score but also by a novel fine-grained taxonomy of ASR-NER errors. Our experiments reveal that encoder-decoder architecture outperforms decoder-only architecture with a short context, while decoder-only architecture benefits from a long context as it fully exploits all layers of the LLM. By using LLM, we significantly reduced the entity omission errors and improved the entity ASR accuracy compared to the Conformer baseline. Additionally, we obtained a state-of-the-art (SOTA) F1 score of 0.805 on the AISHELL-NER test set by using chain-of-thought (CoT) NER which first infers long-form ASR transcriptions and then predicts NER labels.

翻译：将语音令牌映射至与文本令牌相同的特征空间，已成为将语音模态集成至仅解码器大型语言模型的主流范式。另一种替代方法采用编码器-解码器架构，通过交叉注意力机制融合语音特征，然而这一方法在文献中受到的关注较少。本研究将Whisper编码器与ChatGLM3相连接，并针对中文自动语音识别（ASR）与命名实体识别（NER）任务，深入对比了这两种方法。我们不仅采用F1分数等传统评估指标，还引入了一种新型ASR-NER错误细粒度分类体系进行评价。实验表明，在短上下文场景下，编码器-解码器架构优于仅解码器架构；而仅解码器架构则受益于长上下文，因其能充分利用大语言模型的所有层。通过使用大语言模型，与Conformer基线相比，我们显著减少了实体遗漏错误并提升了实体ASR准确率。此外，通过采用思维链（CoT）NER方法——先推断长格式ASR转录文本再预测NER标签——我们在AISHELL-NER测试集上取得了0.805的F1分数，达到当前最优（SOTA）水平。