We describe an approach for aligning an LLM-based dialogue agent based on global (i.e., dialogue-level) rewards, while also taking into account naturally-occurring multimodal signals. At a high level, our approach (dubbed GELI) learns a local, turn-level reward model by decomposing the human-provided Global Explicit (GE) session-level reward, using Local Implicit (LI) multimodal reward signals to crossmodally shape the reward decomposition step. This decomposed reward model is then used as part of the standard RHLF pipeline improve an LLM-based dialog agent. We run quantitative and qualitative human studies to evaluate the performance of our GELI approach, and find that it shows consistent improvements across various conversational metrics compared to baseline methods.

翻译：我们提出了一种方法，在考虑自然产生的多模态信号的同时，基于全局（即对话级别）奖励来对齐基于大语言模型的对话智能体。在高层次上，我们的方法（命名为GELI）通过利用局部隐式多模态奖励信号跨模态塑造奖励分解步骤，学习一个基于人类提供的全局显式会话级别奖励分解的局部逐轮次奖励模型。随后，该分解后的奖励模型作为标准RLHF流程的一部分，用于优化基于大语言模型的对话智能体。我们通过定量和定性人类研究评估GELI方法的性能，发现相比于基线方法，该方法在多种对话评估指标上均展现出持续改进。