Recently, substantial advancements in pre-trained vision-language models have greatly enhanced the capabilities of multi-modal dialog systems. These models have demonstrated significant improvements by fine-tuning on downstream tasks. However, the existing pre-trained models primarily focus on effectively capturing the alignment between vision and language modalities, often ignoring the intricate nature of dialog context. In this paper, we propose a parameter-efficient prompt-tuning method named DialCLIP for multi-modal dialog retrieval. Specifically, our approach introduces a multi-modal context prompt generator to learn context features which are subsequently distilled into prompts within the pre-trained vision-language model CLIP. Besides, we introduce domain prompt to mitigate the disc repancy from the downstream dialog data. To facilitate various types of retrieval, we also design multiple experts to learn mappings from CLIP outputs to multi-modal representation space, with each expert being responsible to one specific retrieval type. Extensive experiments show that DialCLIP achieves state-of-the-art performance on two widely recognized benchmark datasets (i.e., PhotoChat and MMDialog) by tuning a mere 0.04% of the total parameters. These results highlight the efficacy and efficiency of our proposed approach, underscoring its potential to advance the field of multi-modal dialog retrieval.

翻译：近期，预训练视觉语言模型的显著进展极大提升了多模态对话系统的能力，这些模型通过在下游任务上进行微调展现出显著性能改善。然而，现有预训练模型主要聚焦于有效捕捉视觉与语言模态之间的对齐关系，往往忽略对话上下文的复杂特性。本文提出一种名为DialCLIP的参数高效提示调优方法，专门用于多模态对话检索。具体而言，我们的方法引入多模态上下文提示生成器来学习上下文特征，随后将这些特征蒸馏至预训练视觉语言模型CLIP的提示中。此外，我们引入领域提示以缓解下游对话数据带来的不匹配问题。为支持多种检索类型，我们还设计多个专家模块，负责将CLIP输出映射至多模态表示空间，每个专家专注于特定检索类型。大量实验表明，仅需调整总参数的0.04%，DialCLIP即可在两个广泛认可的基准数据集（即PhotoChat和MMDialog）上达到最先进性能。这些结果突显了本方法的有效性与高效性，彰显其在推动多模态对话检索领域发展的潜力。