Visual prosody may be critical for communication success in face-to-face conversations in noisy settings. Here, we explore the involvement of hand, head, and whole-body movements, as well as gesturing quality, in dyadic conversations in noisy settings. We hypothesize that increasing background noise would alter the frequency of conversation-related movements to support the roles of the speaker and the listener. Specifically, talkers may increase gesticulation and thus the use of hand, head, trunk, or leg movements more often, while listeners may increase backchanneling or head and trunk movements to improve the signal-to-noise ratio. Additionally, we test whether the synchrony between speech and hand gestures is affected by background noise. Here, pairs of normal hearing participants (n=8) stood in an audiovisual virtual environment while talking freely. The conversational movements were described using a newly developed labeling system with categories that respect their communicative function. The results showed higher gesturing rate during speaking than during listening. Increased levels of background noise led to increased hand-gesture complexity, modulation of head movements, and a change in trunk movements. People spoke 0.7 dB - 1.4 dB louder during hand gesturing in comparison to times with static drop posture but this was unrelated to presence of background noise. The analysis of hand-speech synchrony showed a modest decrease in synchrony for moderate noise level. People adapt their communicative behavior to increased background noise levels by increases in speech production levels and gesturing which may drive additional increase in speech production due to biomechanical coupling; listeners may increase backchanneling to support the exchange and their own signal-to-noise ratio. The synchrony analysis may reflect motivational factors of communication in noisy environments.

翻译：在嘈杂环境下的面对面对话中，视觉韵律可能对沟通成功至关重要。本文探讨了在噪声环境下的双人对话中，手部、头部及全身运动以及手势质量的参与情况。我们假设背景噪声的增加会改变与对话相关动作的频率，以支持说话者和听者的角色。具体而言，说话者可能增加手势动作，从而更频繁地使用手部、头部、躯干或腿部运动；而听者可能增加反馈信号或头部与躯干运动以提高信噪比。此外，我们测试了语音与手势之间的同步性是否受背景噪声影响。在此实验中，正常听力参与者（n=8）成对站在视听虚拟环境中进行自由对话。对话动作采用新开发的标注系统进行描述，该系统根据动作的交际功能进行分类。结果显示，说话时的手势率高于倾听时。背景噪声水平的增加导致手势复杂度提高、头部运动调节以及躯干运动变化。相较于静态下垂姿势，人们在做出手势时说话音量提高了0.7 dB - 1.4 dB，但这与背景噪声的存在无关。手语同步性分析显示，在中等噪声水平下同步性略有下降。人们通过提高语音产出水平和增加手势来适应增强的背景噪声，而生物力学耦合可能进一步驱动语音产出的增加；听者可能通过增强反馈信号来支持交流并提升自身信噪比。同步性分析可能反映了噪声环境中沟通的动机因素。