Natural language plays a critical role in many computer vision applications, such as image captioning, visual question answering, and cross-modal retrieval, to provide fine-grained semantic information. Unfortunately, while human pose is key to human understanding, current 3D human pose datasets lack detailed language descriptions. To address this issue, we have introduced the PoseScript dataset. This dataset pairs more than six thousand 3D human poses from AMASS with rich human-annotated descriptions of the body parts and their spatial relationships. Additionally, to increase the size of the dataset to a scale that is compatible with data-hungry learning algorithms, we have proposed an elaborate captioning process that generates automatic synthetic descriptions in natural language from given 3D keypoints. This process extracts low-level pose information, known as "posecodes", using a set of simple but generic rules on the 3D keypoints. These posecodes are then combined into higher level textual descriptions using syntactic rules. With automatic annotations, the amount of available data significantly scales up (100k), making it possible to effectively pretrain deep models for finetuning on human captions. To showcase the potential of annotated poses, we present three multi-modal learning tasks that utilize the PoseScript dataset. Firstly, we develop a pipeline that maps 3D poses and textual descriptions into a joint embedding space, allowing for cross-modal retrieval of relevant poses from large-scale datasets. Secondly, we establish a baseline for a text-conditioned model generating 3D poses. Thirdly, we present a learned process for generating pose descriptions. These applications demonstrate the versatility and usefulness of annotated poses in various tasks and pave the way for future research in the field.

翻译：自然语言在图像描述、视觉问答、跨模态检索等众多计算机视觉应用中扮演着关键角色，能够提供细粒度的语义信息。然而，尽管人体姿态对人类理解至关重要，当前3D人体姿态数据集仍缺乏详细的语言描述。为了解决这一问题，我们引入了PoseScript数据集。该数据集将AMASS中超过六千个3D人体姿态与经过人工标注的丰富身体部位及其空间关系描述进行配对。此外，为将数据集规模扩展至能满足数据密集型学习算法的需求，我们提出了一种精细的描述生成流程，该流程能从给定的3D关键点自动生成自然的合成描述。该流程通过一组基于3D关键点的简单通用规则提取底层姿态信息（称为“姿态编码”），再通过句法规则将这些姿态编码组合成更高层次的文本描述。借助自动标注，可用数据量显著提升至10万条，使得深度模型能够在大规模人类描述数据上进行有效预训练并微调。为展示标注姿态的潜力，我们提出了三项利用PoseScript数据集的多模态学习任务。首先，我们构建了一个将3D姿态与文本描述映射到联合嵌入空间的管道，支持从大规模数据集中进行相关姿态的跨模态检索。其次，我们为基于文本条件生成3D姿态的模型建立了基线。最后，我们提出了一种用于生成姿态描述的学习流程。这些应用展示了标注姿态在各种任务中的多功能性与实用性，并为该领域的未来研究奠定了基础。