Accurate prediction of human behavior is crucial for AI systems to effectively support real-world applications, such as autonomous robots anticipating and assisting with human tasks. Real-world scenarios frequently present challenges such as occlusions and incomplete scene observations, which can compromise predictive accuracy. Thus, traditional video-based methods often struggle due to limited temporal and spatial perspectives. Large Language Models (LLMs) offer a promising alternative. Having been trained on a large text corpus describing human behaviors, LLMs likely encode plausible sequences of human actions in a home environment. However, LLMs, trained primarily on text data, lack inherent spatial awareness and real-time environmental perception. They struggle with understanding physical constraints and spatial geometry. Therefore, to be effective in a real-world spatial scenario, we propose a multimodal prediction framework that enhances LLM-based action prediction by integrating physical constraints derived from human trajectories. Our experiments demonstrate that combining LLM predictions with trajectory data significantly improves overall prediction performance. This enhancement is particularly notable in situations where the LLM receives limited scene information, highlighting the complementary nature of linguistic knowledge and physical constraints in understanding and anticipating human behavior.

翻译：准确预测人类行为对于AI系统有效支持现实世界应用至关重要，例如使自主机器人能够预判并协助人类任务。现实场景常面临遮挡与不完整场景观测等挑战，这些因素可能损害预测准确性。因此，传统基于视频的方法常因时空视角有限而难以应对。大语言模型（LLMs）为此提供了有前景的替代方案。LLMs经过大规模描述人类行为的文本语料训练，可能编码了家庭环境中合理的人类动作序列。然而，主要基于文本数据训练的LLMs缺乏固有的空间意识与实时环境感知能力，在理解物理约束与空间几何关系方面存在局限。为此，我们提出一种多模态预测框架，通过整合从人类轨迹推导的物理约束来增强基于LLM的行为预测，使其在现实空间场景中有效发挥作用。实验表明，将LLM预测与轨迹数据相结合能显著提升整体预测性能。这种增强在LLM获取有限场景信息的情况下尤为显著，凸显了语言知识与物理约束在理解和预测人类行为方面的互补性。