While Large Language Models (LLMs) have demonstrated commendable performance across a myriad of domains and tasks, existing LLMs still exhibit a palpable deficit in handling multimodal functionalities, especially for the Spoken Question Answering (SQA) task which necessitates precise alignment and deep interaction between speech and text features. To address the SQA challenge on LLMs, we initially curated the free-form and open-ended LibriSQA dataset from Librispeech, comprising Part I with natural conversational formats and Part II encompassing multiple-choice questions followed by answers and analytical segments. Both parts collectively include 107k SQA pairs that cover various topics. Given the evident paucity of existing speech-text LLMs, we propose a lightweight, end-to-end framework to execute the SQA task on the LibriSQA, witnessing significant results. By reforming ASR into the SQA format, we further substantiate our framework's capability in handling ASR tasks. Our empirical findings bolster the LLMs' aptitude for aligning and comprehending multimodal information, paving the way for the development of universal multimodal LLMs. The dataset and demo can be found at https://github.com/ZihanZhaoSJTU/LibriSQA.

翻译：尽管大语言模型（LLMs）在众多领域和任务中展现了卓越性能，但现有LLMs在处理多模态功能方面仍存在明显不足，尤其是在需要语音与文本特征精准对齐与深度交互的口语问答（SQA）任务上。为应对LLMs面临的SQA挑战，我们首先从Librispeech中构建了自由形式与开放式的LibriSQA数据集，其中第一部分包含自然对话格式，第二部分涵盖多选题及其答案与分析片段。两部分共包含107k个涵盖多种主题的SQA对。鉴于现有语音-文本LLMs明显匮乏，我们提出一种轻量级端到端框架在LibriSQA上执行SQA任务，并取得了显著成果。通过将ASR任务重构为SQA格式，我们进一步验证了该框架处理ASR任务的能力。我们的实证结果增强了LLMs对齐与理解多模态信息的能力，为通用多模态LLMs的开发铺平了道路。数据集和演示可在 https://github.com/ZihanZhaoSJTU/LibriSQA 获取。