Sarcasm, sentiment, and emotion are three typical kinds of spontaneous affective responses of humans to external events and they are tightly intertwined with each other. Such events may be expressed in multiple modalities (e.g., linguistic, visual and acoustic), e.g., multi-modal conversations. Joint analysis of humans' multi-modal sarcasm, sentiment, and emotion is an important yet challenging topic, as it is a complex cognitive process involving both cross-modality interaction and cross-affection correlation. From the probability theory perspective, cross-affection correlation also means that the judgments on sarcasm, sentiment, and emotion are incompatible. However, this exposed phenomenon cannot be sufficiently modelled by classical probability theory due to its assumption of compatibility. Neither do the existing approaches take it into consideration. In view of the recent success of quantum probability (QP) in modeling human cognition, particularly contextual incompatible decision making, we take the first step towards introducing QP into joint multi-modal sarcasm, sentiment, and emotion analysis. Specifically, we propose a QUantum probabIlity driven multi-modal sarcasm, sEntiment and emoTion analysis framework, termed QUIET. Extensive experiments on two datasets and the results show that the effectiveness and advantages of QUIET in comparison with a wide range of the state-of-the-art baselines. We also show the great potential of QP in multi-affect analysis.

翻译：讽刺、情感和情绪是人类对外部事件的三种典型自发情感反应，且三者紧密交织。这类事件可能以多种模态（如语言、视觉和听觉）呈现，例如多模态对话。联合分析人类的多模态讽刺、情感与情绪是一个重要但极具挑战性的课题，这涉及跨模态交互与跨情感关联的复杂认知过程。从概率论视角看，跨情感关联也意味着对讽刺、情感和情绪的判定具有不相容性。然而，这一现象因传统概率论对相容性的假设而无法得到充分建模，现有方法也均未考虑该问题。鉴于量子概率在人类认知建模（尤其是上下文不相容决策）领域的最新成功，我们首次将量子概率引入多模态讽刺、情感与情绪的联合分析。具体而言，我们提出了一种量子概率驱动的多模态讽刺、情感与情绪分析框架，命名为QUIET。在两个数据集上的广泛实验结果表明，与各类最新基线方法相比，QUIET具有显著有效性和优越性。我们还展示了量子概率在多情感分析中的巨大潜力。