Simultaneous interpretation (SI), the translation of one language to another in real time, starts translation before the original speech has finished. Its evaluation needs to consider both latency and quality. This trade-off is challenging especially for distant word order language pairs such as English and Japanese. To handle this word order gap, interpreters maintain the word order of the source language as much as possible to keep up with original language to minimize its latency while maintaining its quality, whereas in translation reordering happens to keep fluency in the target language. This means outputs synchronized with the source language are desirable based on the real SI situation, and it's a key for further progress in computational SI and simultaneous machine translation (SiMT). In this work, we propose an automatic evaluation metric for SI and SiMT focusing on word order synchronization. Our evaluation metric is based on rank correlation coefficients, leveraging cross-lingual pre-trained language models. Our experimental results on NAIST-SIC-Aligned and JNPC showed our metrics' effectiveness to measure word order synchronization between source and target language.

翻译：同声传译（SI）是一种将一种语言实时翻译成另一种语言的过程，其在原语发言结束前即开始翻译。其评估需同时考虑延迟和质量。这种权衡对于词序差异较大的语言对（如英语和日语）尤其具有挑战性。为处理这种词序差异，译员在保持质量的同时，会尽可能遵循源语言的词序以跟上原语节奏、最小化延迟，而在笔译中则会发生词序重组以保持目标语言的流畅性。这意味着基于真实同声传译场景，与源语言同步的输出是理想的，这也是计算同声传译和同步机器翻译（SiMT）取得进一步进展的关键。在本工作中，我们提出了一种专注于词序同步的同声传译与同步机器翻译自动评估指标。我们的评估指标基于秩相关系数，并利用了跨语言预训练语言模型。我们在NAIST-SIC-Aligned和JNPC数据集上的实验结果表明，该指标能有效度量源语言与目标语言之间的词序同步程度。