Social interactions dominate our perceptions of the world and shape our daily behavior by attaching social meaning to acts as simple and spontaneous as gestures, facial expressions, voice, and speech. People mimic and otherwise respond to each other's postures, facial expressions, mannerisms, and other verbal and nonverbal behavior, and form appraisals or evaluations in the process. Yet, no publicly-available dataset includes multimodal recordings and self-report measures of multiple persons in social interaction. Dyadic recordings and annotation are lacking. We present a new data corpus of multimodal dyadic interaction (45 dyads, 90 persons) that includes synchronized multi-modality behavior (2D face video, 3D face geometry, thermal spectrum dynamics, voice and speech behavior, physiology (PPG, EDA, heart-rate, blood pressure, and respiration), and self-reported affect of all participants in a communicative interaction scenario. Two types of dyads are included: persons with shared past history and strangers. Annotations include social signals, agreement, disagreement, and neutral stance. With a potent emotion induction, these multimodal data will enable novel modeling of multimodal interpersonal behavior. We present extensive experiments to evaluate multimodal dyadic communication of dyads with and without interpersonal history, and their affect. This new database will make multimodal modeling of social interaction never possible before. The dataset includes 20TB of multimodal data to share with the research community.

翻译：社交互动主导着我们对世界的感知，并通过赋予手势、面部表情、声音和言语等简单而自发的行为以社会意义，塑造着我们的日常行为。人们会模仿并回应彼此的姿态、面部表情、举止以及其他言语和非言语行为，并在此过程中形成认知或评价。然而，目前尚无公开数据集包含多人社交互动中的多模态记录和自我报告测量，尤其是双人互动的记录和标注仍属空白。我们提出了一个多模态双人互动新语料库（45组双人互动，共90人），包含同步的多模态行为数据（二维面部视频、三维面部几何、热谱动态、声音与言语行为、生理信号（PPG、EDA、心率、血压和呼吸）），以及所有参与者在交流互动场景中的自我报告情感。语料库包含两类双人互动：具有共同过往经历者与陌生人。标注内容包括社交信号、同意、不同意及中立立场。通过有效的情感诱导，这些多模态数据将支持对多模态人际行为进行创新性建模。我们开展了大量实验，以评估有无人际交往史的双人互动多模态沟通及其情感表现。这一新数据库将首次实现对社交互动的多模态建模。该数据集包含20TB多模态数据，可供研究社区共享使用。