Emotion detection presents challenges to intelligent human-robot interaction (HRI). Foundational deep learning techniques used in emotion detection are limited by information-constrained datasets or models that lack the necessary complexity to learn interactions between input data elements, such as the the variance of human emotions across different contexts. In the current effort, we introduce 1) MoEmo (Motion to Emotion), a cross-attention vision transformer (ViT) for human emotion detection within robotics systems based on 3D human pose estimations across various contexts, and 2) a data set that offers full-body videos of human movement and corresponding emotion labels based on human gestures and environmental contexts. Compared to existing approaches, our method effectively leverages the subtle connections between movement vectors of gestures and environmental contexts through the use of cross-attention on the extracted movement vectors of full-body human gestures/poses and feature maps of environmental contexts. We implement a cross-attention fusion model to combine movement vectors and environment contexts into a joint representation to derive emotion estimation. Leveraging our Naturalistic Motion Database, we train the MoEmo system to jointly analyze motion and context, yielding emotion detection that outperforms the current state-of-the-art.

翻译：情感检测为智能人机交互（HRI）带来挑战。现有用于情感检测的基础深度学习技术受限于信息受限的数据集或模型，这些模型缺乏学习输入数据元素间交互所需复杂度（例如人类情感在不同情境中的变异性）。在本研究中，我们提出：1）MoEmo（动作到情感）——一种基于不同情境下3D人体姿态估计的交叉注意力视觉Transformer（ViT），用于机器人系统内的人类情感检测；2）一个提供全身人体运动视频及基于手势与环境情境的情感标签的数据集。与现有方法相比，我们的方法通过提取全身手势/姿态运动向量与环境情境特征图上的交叉注意力机制，有效利用手势运动向量与环境情境间的微妙关联。我们实现了一种交叉注意力融合模型，将运动向量与环境情境结合为联合表征以推导情感估计。借助自然动作数据库，我们训练MoEmo系统联合分析动作与情境，使情感检测性能超越当前最优水平。