Video-grounded Dialogue (VGD) aims to answer questions regarding a given multi-modal input comprising video, audio, and dialogue history. Although there have been numerous efforts in developing VGD systems to improve the quality of their responses, existing systems are competent only to incorporate the information in the video and text and tend to struggle in extracting the necessary information from the audio when generating appropriate responses to the question. The VGD system seems to be deaf, and thus, we coin this symptom of current systems' ignoring audio data as a deaf response. To overcome the deaf response problem, Hearing Enhanced Audio Response (HEAR) framework is proposed to perform sensible listening by selectively attending to audio whenever the question requires it. The HEAR framework enhances the accuracy and audibility of VGD systems in a model-agnostic manner. HEAR is validated on VGD datasets (i.e., AVSD@DSTC7 and AVSD@DSTC8) and shows effectiveness with various VGD systems.

翻译：视频对话（VGD）旨在回答关于包含视频、音频和对话历史的多模态输入的问题。尽管已有大量研究致力于开发VGD系统以提升其回复质量，但现有系统仅能有效整合视频与文本中的信息，在生成恰当回复时往往难以从音频中提取必要信息。这种VGD系统如同"失聪"般忽略音频数据的现象，我们称之为"聋态响应"。为克服聋态响应问题，本文提出听觉增强音频响应（HEAR）框架，该框架通过选择性关注问题所需的音频信息实现智能聆听，以模型无关的方式提升VGD系统的准确性与可听性。HEAR在VGD数据集（即AVSD@DSTC7与AVSD@DSTC8）上经过验证，证明其在不同VGD系统中的有效性。