Vision-language models are increasingly employed as multimodal conversational agents (MCAs) for diverse conversational tasks. Recently, reinforcement learning (RL) has been widely explored for adapting MCAs to various human-AI interaction scenarios. Despite showing great enhancement in generalization performance, fine-tuning MCAs via RL still faces challenges in handling the extremely large text token space. To address this, we learn a compact latent action space for RL fine-tuning instead. Specifically, we adopt the learning from observation mechanism to construct the codebook for the latent action space, where future observations are leveraged to estimate current latent actions that could further be used to reconstruct future observations. However, the scarcity of paired image-text data hinders learning a codebook with sufficient coverage. Thus, we leverage both paired image-text data and text-only data to construct the latent action space, using a cross-modal projector for transforming text embeddings into image-text embeddings. We initialize the cross-modal projector on paired image-text data, and further train it on massive text-only data with a novel cycle consistency loss to enhance its robustness. We show that our latent action based method outperforms competitive baselines on two conversation tasks across various RL algorithms.

翻译：视觉语言模型正日益广泛地作为多模态对话智能体应用于多样化的对话任务。近期，强化学习被广泛探索用于使多模态对话智能体适应各种人机交互场景。尽管在泛化性能上展现出显著提升，通过强化学习对多模态对话智能体进行微调在处理极大文本标记空间方面仍面临挑战。为解决此问题，我们转而学习一个紧凑的隐式动作空间用于强化学习微调。具体而言，我们采用从观察中学习的机制来构建隐式动作空间的码本，其中利用未来观察来估计当前的隐式动作，这些动作可进一步用于重建未来观察。然而，配对图文数据的稀缺性阻碍了学习具有充分覆盖度的码本。因此，我们同时利用配对图文数据与纯文本数据来构建隐式动作空间，使用一个跨模态投影器将文本嵌入转换为图文嵌入。我们在配对图文数据上初始化跨模态投影器，并进一步在大量纯文本数据上使用新颖的循环一致性损失进行训练，以增强其鲁棒性。实验表明，我们基于隐式动作的方法在两种对话任务上，相较于多种强化学习算法的竞争基线，均取得了更优的性能。