Cross-modal retrieval has become a prominent research topic in computer vision and natural language processing with advances made in image-text and video-text retrieval technologies. However, cross-modal retrieval between human motion sequences and text has not garnered sufficient attention despite the extensive application value it holds, such as aiding virtual reality applications in better understanding users' actions and language. This task presents several challenges, including joint modeling of the two modalities, demanding the understanding of person-centered information from text, and learning behavior features from 3D human motion sequences. Previous work on motion data modeling mainly relied on autoregressive feature extractors that may forget previous information, while we propose an innovative model that includes simple yet powerful transformer-based motion and text encoders, which can learn representations from the two different modalities and capture long-term dependencies. Furthermore, the overlap of the same atomic actions of different human motions can cause semantic conflicts, leading us to explore a new triplet loss function, MildTriple Loss. it leverages the similarity between samples in intra-modal space to guide soft-hard negative sample mining in the joint embedding space to train the triplet loss and reduce the violation caused by false negative samples. We evaluated our model and method on the latest HumanML3D and KIT Motion-Language datasets, achieving a 62.9\% recall for motion retrieval and a 71.5\% recall for text retrieval (based on R@10) on the HumanML3D dataset. Our code is available at https://github.com/eanson023/rehamot.

翻译：跨模态检索作为计算机视觉与自然语言处理领域的重要研究方向，已在图像-文本和视频-文本检索技术方面取得显著进展。然而，尽管人体运动序列与文本之间的跨模态检索在虚拟现实等应用中具有重要价值（例如帮助系统理解用户的行为与语言），该方向尚未获得足够关注。该任务面临多项挑战，包括两种模态的联合建模、从文本中理解人物中心信息、以及从三维人体运动序列中学习行为特征。以往的运动数据建模主要依赖可能遗忘先前信息的自回归特征提取器，而本文提出了一种创新模型，包含简洁而强大的基于Transformer的运动与文本编码器，既能从两种不同模态中学习表征，又能捕获长期依赖关系。此外，不同人体运动中相同原子动作的重叠可能导致语义冲突，为此我们探索了新型三元组损失函数MildTriple Loss。该函数利用模态内样本间的相似度指导联合嵌入空间中的软硬负样本挖掘，从而训练三元组损失并减少假阴性样本造成的违反。我们在最新的HumanML3D和KIT运动-语言数据集上评估了模型与方法，在HumanML3D数据集上实现了62.9%的运动检索召回率和71.5%的文本检索召回率（基于R@10）。代码已开源至https://github.com/eanson023/rehamot。