Multimodal data analysis and validation based on streams from state-of-the-art sensor technology such as eye-tracking or emotion recognition using the Facial Action Coding System (FACTs) with deep learning allows educational researchers to study multifaceted learning and problem-solving processes and to improve educational experiences. This study aims to investigate the correlation between two continuous sensor streams, pupil diameter as an indicator of cognitive workload and FACTs with deep learning as an indicator of emotional arousal (RQ 1a), specifically for epochs of high, medium, and low arousal (RQ 1b). Furthermore, the time lag between emotional arousal and pupil diameter data will be analyzed (RQ 2). 28 participants worked on three cognitively demanding and emotionally engaging everyday moral dilemmas while eye-tracking and emotion recognition data were collected. The data were pre-processed in Phyton (synchronization, blink control, downsampling) and analyzed using correlation analysis and Granger causality tests. The results show negative and statistically significant correlations between the data streams for emotional arousal and pupil diameter. However, the correlation is negative and significant only for epochs of high arousal, while positive but non-significant relationships were found for epochs of medium or low arousal. The average time lag for the relationship between arousal and pupil diameter was 2.8 ms. In contrast to previous findings without a multimodal approach suggesting a positive correlation between the constructs, the results contribute to the state of research by highlighting the importance of multimodal data validation and research on convergent vagility. Future research should consider emotional regulation strategies and emotional valence.

翻译：多模态数据分析与验证基于眼动追踪等前沿传感器技术流，以及采用深度学习的面部动作编码系统（FACTs）进行情绪识别，使教育研究者能够研究多层面的学习与问题解决过程，并改善教育体验。本研究旨在探究两个连续传感器流之间的相关性：作为认知负荷指标的瞳孔直径，与作为情绪唤醒指标的深度学习FACTs（研究问题1a），特别针对高、中、低唤醒时段（研究问题1b）。此外，还将分析情绪唤醒与瞳孔直径数据之间的时间滞后（研究问题2）。28名参与者在完成三项认知要求高且情感投入的日常道德困境任务时，同步收集了眼动追踪和情绪识别数据。数据在Python中预处理（同步、眨眼控制、降采样），并采用相关分析和格兰杰因果检验进行分析。结果显示，情绪唤醒与瞳孔直径数据流之间存在负向且统计显著的相关性。然而，仅在高唤醒时段呈现负向显著相关，而在中、低唤醒时段则发现正向但不显著的关系。情绪唤醒与瞳孔直径之间的平均时间滞后为2.8毫秒。与先前未采用多模态方法但表明两者正相关的研究结论不同，本研究结果通过强调多模态数据验证及收敛效度研究的重要性，为现有研究领域做出了贡献。未来研究应考虑情绪调节策略和情绪效价。