Cross-modal retrieval has become a prominent research topic in computer vision and natural language processing with advances made in image-text and video-text retrieval technologies. However, cross-modal retrieval between human motion sequences and text has not garnered sufficient attention despite the extensive application value it holds, such as aiding virtual reality applications in better understanding users' actions and language. This task presents several challenges, including joint modeling of the two modalities, demanding the understanding of person-centered information from text, and learning behavior features from 3D human motion sequences. Previous work on motion data modeling mainly relied on autoregressive feature extractors that may forget previous information, while we propose an innovative model that includes simple yet powerful transformer-based motion and text encoders, which can learn representations from the two different modalities and capture long-term dependencies. Furthermore, the overlap of the same atomic actions of different human motions can cause semantic conflicts, leading us to explore a new triplet loss function, MildTriple Loss. it leverages the similarity between samples in intra-modal space to guide soft-hard negative sample mining in the joint embedding space to train the triplet loss and reduce the violation caused by false negative samples. We evaluated our model and method on the latest HumanML3D and KIT Motion-Language datasets, achieving a 62.9\% recall for motion retrieval and a 71.5\% recall for text retrieval (based on R@10) on the HumanML3D dataset. Our code is available at https://github.com/eanson023/rehamot.

翻译：跨模态检索已成为计算机视觉和自然语言处理领域的重要研究方向，在图像-文本和视频-文本检索技术方面取得了显著进展。然而，人体运动序列与文本之间的跨模态检索尚未获得足够关注，尽管其在辅助虚拟现实应用理解用户动作与语言等方面具有广泛的应用价值。该任务面临多重挑战，包括双模态的联合建模、从文本中理解以人为中心的信息、以及从三维人体运动序列中学习行为特征。先前关于运动数据建模的研究主要依赖自回归特征提取器，这类方法易遗忘历史信息。为此，我们提出创新模型，采用简洁而强大的基于Transformer的运动编码器和文本编码器，既能学习两种不同模态的表征，又能捕捉长期依赖关系。此外，不同人体动作中相同原子动作的重叠可能导致语义冲突，为此我们探索了新型三元组损失函数MildTriple Loss。该函数利用模态内样本间的相似性指导联合嵌入空间中的软-硬负样本挖掘，从而训练三元组损失并减少伪负样本造成的违反。我们在最新的HumanML3D和KIT运动-语言数据集上评估了模型与方法，在HumanML3D数据集上分别实现了62.9%的动作检索召回率和71.5%的文本检索召回率（基于R@10指标）。我们的代码开源在https://github.com/eanson023/rehamot。