In this paper, we investigate building a sequence to sequence architecture for motion to language translation and synchronization. The aim is to translate motion capture inputs into English natural-language descriptions, such that the descriptions are generated synchronously with the actions performed, enabling semantic segmentation as a byproduct, but without requiring synchronized training data. We propose a new recurrent formulation of local attention that is suited for synchronous/live text generation, as well as an improved motion encoder architecture better suited to smaller data and for synchronous generation. We evaluate both contributions in individual experiments, using the standard BLEU4 metric, as well as a simple semantic equivalence measure, on the KIT motion language dataset. In a follow-up experiment, we assess the quality of the synchronization of generated text in our proposed approaches through multiple evaluation metrics. We find that both contributions to the attention mechanism and the encoder architecture additively improve the quality of generated text (BLEU and semantic equivalence), but also of synchronization. Our code is available at https://github.com/rd20karim/M2T-Segmentation/tree/main

翻译：本文研究了构建用于运动到语言翻译与同步化的序列到序列架构。目标是动作捕捉输入翻译为英文自然语言描述，并使描述与执行的动作同步生成，从而作为副产品实现语义分割，且无需同步训练数据。我们提出了一种适用于同步/实时文本生成的局部注意力递归公式，以及一种改进的运动编码器架构，该架构更适合小规模数据和同步生成。通过标准BLEU4指标和简单的语义等价度量，我们在KIT运动语言数据集上分别评估了这两项贡献。在后续实验中，我们通过多项评估指标衡量所提方法生成文本的同步质量。研究发现，注意力机制和编码器架构的改进在提升生成文本质量（BLEU和语义等价性）的同时，也改善了同步性能。我们的代码开源于 https://github.com/rd20karim/M2T-Segmentation/tree/main