We introduce "Generative Fusion Decoding" (GFD), a novel shallow fusion framework, utilized to integrate Large Language Models (LLMs) into multi-modal text recognition systems such as automatic speech recognition (ASR) and optical character recognition (OCR). We derive the formulas necessary to enable GFD to operate across mismatched token spaces of different models by mapping text token space to byte token space, enabling seamless fusion during the decoding process. The framework is plug-and-play, compatible with various auto-regressive models, and does not require re-training for feature alignment, thus overcoming limitations of previous fusion techniques. We highlight three main advantages of GFD: First, by simplifying the complexity of aligning different model sample spaces, GFD allows LLMs to correct errors in tandem with the recognition model, reducing computation latencies. Second, the in-context learning ability of LLMs is fully capitalized by GFD, increasing robustness in long-form speech recognition and instruction aware speech recognition. Third, GFD enables fusing recognition models deficient in Chinese text recognition with LLMs extensively trained on Chinese. Our evaluation demonstrates that GFD significantly improves performance in ASR and OCR tasks, with ASR reaching state-of-the-art in the NTUML2021 benchmark. GFD provides a significant step forward in model integration, offering a unified solution that could be widely applicable to leveraging existing pre-trained models through step by step fusion.

翻译：本文提出“生成式融合解码”（GFD）这一新颖的浅层融合框架，用于将大语言模型（LLMs）集成到自动语音识别（ASR）和光学字符识别（OCR）等多模态文本识别系统中。我们推导了使GFD能够在不同模型的不匹配标记空间上运行的必要公式，通过将文本标记空间映射到字节标记空间，实现解码过程中的无缝融合。该框架即插即用，兼容多种自回归模型，且无需为特征对齐重新训练，从而克服了以往融合技术的局限性。我们重点阐述了GFD的三个主要优势：首先，通过简化不同模型采样空间的对齐复杂度，GFD使LLMs能够与识别模型协同纠正错误，降低计算延迟。其次，GFD充分利用了LLMs的上下文学习能力，增强了长语音识别和指令感知语音识别的鲁棒性。第三，GFD使得在中文文本识别能力不足的识别模型与经过大量中文训练的大语言模型之间实现融合成为可能。评估结果表明，GFD显著提升了ASR和OCR任务的性能，其中ASR在NTUML2021基准测试中达到了最先进水平。GFD在模型集成方面迈出了重要一步，提供了一种统一的解决方案，有望通过逐步融合的方式广泛应用于现有预训练模型的利用。