We introduce a video framework for modeling the association between verbal and non-verbal communication during dyadic conversation. Given the input speech of a speaker, our approach retrieves a video of a listener, who has facial expressions that would be socially appropriate given the context. Our approach further allows the listener to be conditioned on their own goals, personalities, or backgrounds. Our approach models conversations through a composition of large language models and vision-language models, creating internal representations that are interpretable and controllable. To study multimodal communication, we propose a new video dataset of unscripted conversations covering diverse topics and demographics. Experiments and visualizations show our approach is able to output listeners that are significantly more socially appropriate than baselines. However, many challenges remain, and we release our dataset publicly to spur further progress. See our website for video results, data, and code: https://realtalk.cs.columbia.edu.

翻译：我们提出了一种视频框架，用于建模二元对话过程中言语与非言语交流之间的关联。给定说话者的语音输入，我们的方法能够检索出倾听者的视频，其面部表情在上下文中具有社会适宜性。该方法进一步允许倾听者根据其自身目标、个性或背景进行条件化调节。通过组合大语言模型与视觉语言模型，我们建模了对话过程，生成了可解释且可控的内部表征。为研究多模态交流，我们提出了一个新的非脚本对话视频数据集，涵盖多样化话题与人口统计学特征。实验与可视化结果表明，我们的方法能输出比基线方法显著更具社会适宜性的倾听者。然而，诸多挑战依然存在，我们公开了该数据集以促进进一步研究。视频结果、数据及代码详见我们的网站：https://realtalk.cs.columbia.edu。