We introduce ``Generative Fusion Decoding'' (GFD), a novel shallow fusion framework, utilized to integrate Large Language Models (LLMs) into multi-modal text recognition systems such as automatic speech recognition (ASR) and optical character recognition (OCR). We derive the formulas necessary to enable GFD to operate across mismatched token spaces of different models by mapping text token space to byte token space, enabling seamless fusion during the decoding process. The framework is plug-and-play, compatible with various auto-regressive models, and does not require re-training for feature alignment, thus overcoming limitations of previous fusion techniques. We highlight three main advantages of GFD: First, by simplifying the complexity of aligning different model sample spaces, GFD allows LLMs to correct errors in tandem with the recognition model, reducing computation latencies. Second, the in-context learning ability of LLMs is fully capitalized by GFD, increasing robustness in long-form speech recognition and instruction aware speech recognition. Third, GFD enables fusing recognition models deficient in Chinese text recognition with LLMs extensively trained on Chinese. Our evaluation demonstrates that GFD significantly improves performance in ASR and OCR tasks, with ASR reaching state-of-the-art in the NTUML2021 benchmark. GFD provides a significant step forward in model integration, offering a unified solution that could be widely applicable to leveraging existing pre-trained models through step by step fusion.

翻译：本文提出了一种新颖的浅层融合框架——“生成式融合解码”（Generative Fusion Decoding，GFD），用于将大语言模型（LLMs）集成到自动语音识别（ASR）和光学字符识别（OCR）等多模态文本识别系统中。我们推导了必要的公式，通过将文本标记空间映射到字节标记空间，使GFD能够在不同模型的不匹配标记空间上运行，从而在解码过程中实现无缝融合。该框架即插即用，兼容各种自回归模型，且无需为特征对齐进行重新训练，从而克服了先前融合技术的局限性。我们重点阐述了GFD的三个主要优势：首先，通过简化对齐不同模型采样空间的复杂性，GFD允许LLMs与识别模型协同纠正错误，降低了计算延迟。其次，GFD充分利用了LLMs的上下文学习能力，增强了长语音识别和指令感知语音识别的鲁棒性。第三，GFD能够将中文文本识别能力不足的识别模型与经过大量中文训练的大语言模型进行融合。我们的评估表明，GFD显著提升了ASR和OCR任务的性能，其中ASR在NTUML2021基准测试中达到了最先进水平。GFD在模型集成方面迈出了重要一步，提供了一种统一的解决方案，有望通过逐步融合的方式广泛应用于利用现有预训练模型。