Accurate human trajectory prediction is crucial for applications such as autonomous vehicles, robotics, and surveillance systems. Yet, existing models often fail to fully leverage the non-verbal social cues human subconsciously communicate when navigating the space. To address this, we introduce Social-Transmotion, a generic model that exploits the power of transformers to handle diverse and numerous visual cues, capturing the multi-modal nature of human behavior. We translate the idea of a prompt from Natural Language Processing (NLP) to the task of human trajectory prediction, where a prompt can be a sequence of x-y coordinates on the ground, bounding boxes or body poses. This, in turn, augments trajectory data, leading to enhanced human trajectory prediction. Our model exhibits flexibility and adaptability by capturing spatiotemporal interactions between pedestrians based on the available visual cues, whether they are poses, bounding boxes, or a combination thereof. By the masking technique, we ensure our model's effectiveness even when certain visual cues are unavailable, although performance is further boosted with the presence of comprehensive visual data. We delve into the merits of using 2d versus 3d poses, and a limited set of poses. Additionally, we investigate the spatial and temporal attention map to identify which keypoints and frames of poses are vital for optimizing human trajectory prediction. Our approach is validated on multiple datasets, including JTA, JRDB, Pedestrians and Cyclists in Road Traffic, and ETH-UCY. The code is publicly available: https://github.com/vita-epfl/social-transmotion

翻译：准确的人类轨迹预测对于自动驾驶、机器人技术和监控系统等应用至关重要。然而，现有模型往往未能充分利用人类在空间中移动时下意识传达的非语言社交线索。为解决这一问题，我们提出Social-Transmotion——一个通用模型，它利用Transformer的强大能力处理多样且丰富的视觉线索，捕捉人类行为的多模态特性。我们将自然语言处理（NLP）中的"提示"概念迁移至人类轨迹预测任务：提示可以是地面上的x-y坐标序列、边界框或身体姿态。这一方法通过增强轨迹数据提升了人类轨迹预测的性能。基于可用的视觉线索（无论是姿态、边界框或其组合），我们的模型通过捕捉行人间的时空交互展现出灵活性与适应性。通过掩码技术，我们确保模型在缺失某些视觉线索时依然有效，尽管完整视觉数据的存在会进一步优化性能。我们深入比较了使用2D姿态与3D姿态、以及有限姿态集的效果优劣。此外，通过分析空间和时间注意力图，我们识别出对优化人类轨迹预测至关重要的关键点与姿态帧序列。该方法在JTA、JRDB、道路交通行人与骑行者数据集以及ETH-UCY等多个数据集上得到验证。代码已公开：https://github.com/vita-epfl/social-transmotion