The rise of Large Language Models (LLMs) has affected various disciplines that got beyond mere text generation. Going beyond their textual nature, this project proposal aims to investigate the interaction between LLMs and non-verbal communication, specifically focusing on gestures. The proposal sets out a plan to examine the proficiency of LLMs in deciphering both explicit and implicit non-verbal cues within textual prompts and their ability to associate these gestures with various contextual factors. The research proposes to test established psycholinguistic study designs to construct a comprehensive dataset that pairs textual prompts with detailed gesture descriptions, encompassing diverse regional variations, and semantic labels. To assess LLMs' comprehension of gestures, experiments are planned, evaluating their ability to simulate human behaviour in order to replicate psycholinguistic experiments. These experiments consider cultural dimensions and measure the agreement between LLM-identified gestures and the dataset, shedding light on the models' contextual interpretation of non-verbal cues (e.g. gestures).

翻译：大型语言模型（LLMs）的兴起已影响众多超越单纯文本生成的学科。本课题提案旨在超越其文本本质，探究LLMs与非语言沟通之间的交互，尤其聚焦于手势。提案制定了一项计划，以检验LLMs在文本提示中解读显性与隐性非语言线索的能力，以及将这些手势与各类情境因素关联的能力。研究拟采用已建立的心理语言学实验设计，构建一个全面数据集，将文本提示与详细的手势描述配对，涵盖多样的地域差异及语义标签。为评估LLMs对手势的理解，计划开展实验，评估其模拟人类行为以复现心理语言学实验的能力。这些实验考虑文化维度，并衡量LLMs识别的手势与数据集之间的一致性，从而揭示模型对非语言线索（如手势）的语境解读。