Previous studies regarding the perception of emotions for embodied virtual agents have shown the effectiveness of using virtual characters in conveying emotions through interactions with humans. However, creating an autonomous embodied conversational agent with expressive behaviors presents two major challenges. The first challenge is the difficulty of synthesizing the conversational behaviors for each modality that are as expressive as real human behaviors. The second challenge is that the affects are modeled independently, which makes it difficult to generate multimodal responses with consistent emotions across all modalities. In this work, we propose a conceptual framework, ACTOR (Affect-Consistent mulTimodal behaviOR generation), that aims to increase the perception of affects by generating multimodal behaviors conditioned on a consistent driving affect. We have conducted a user study with 199 participants to assess how the average person judges the affects perceived from multimodal behaviors that are consistent and inconsistent with respect to a driving affect. The result shows that among all model conditions, our affect-consistent framework receives the highest Likert scores for the perception of driving affects. Our statistical analysis suggests that making a modality affect-inconsistent significantly decreases the perception of driving affects. We also observe that multimodal behaviors conditioned on consistent affects are more expressive compared to behaviors with inconsistent affects. Therefore, we conclude that multimodal emotion conditioning and affect consistency are vital to enhancing the perception of affects for embodied conversational agents.

翻译：以往关于具身虚拟代理情感感知的研究表明，通过虚拟角色与人类互动传递情感具有有效性。然而，构建具有表达性行为的自主具身会话代理面临两大挑战：首先，难以合成与真实人类行为同等级别表现力的各模态对话行为；其次，情感建模的独立性导致难以生成跨模态情感一致的多模态响应。本研究提出概念框架ACTOR（情感一致多模态行为生成），旨在通过生成基于一致驱动情感的多模态行为提升情感感知效能。我们开展包含199名参与者的用户研究，评估普通人如何判断与驱动情感一致或不一致的多模态行为所传递的情感。结果显示，在所有模型条件中，我们提出的情感一致框架在驱动情感感知的Likert评分中得分最高。统计分析表明，使某一模态情感不一致会显著降低驱动情感的感知程度。我们还发现，基于一致情感的多模态行为比情感不一致的行为更具表现力。因此得出结论：多模态情感条件作用与情感一致性对于增强具身会话代理的情感感知至关重要。