Simultaneous translation is a task in which the translation begins before the end of an input speech segment. Its evaluation should be conducted based on latency in addition to quality, and for users, the smallest possible amount of latency is preferable. Most existing metrics measure latency based on the start timings of partial translations and ignore their duration. This means such metrics do not penalize the latency caused by long translation output, which delays the comprehension of users and subsequent translations. In this work, we propose a novel latency evaluation metric for simultaneous translation called \emph{Average Token Delay} (ATD) that focuses on the duration of partial translations. We demonstrate its effectiveness through analyses simulating user-side latency based on Ear-Voice Span (EVS). In our experiment, ATD had the highest correlation with EVS among baseline latency metrics under most conditions.
翻译:同声传译是指在输入语音片段结束前即开始翻译的任务。其评估需在质量基础上结合延迟进行,且对用户而言,延迟越小越好。现有大多数指标基于部分翻译的起始时间衡量延迟,忽略了其持续时间。这意味着此类指标不会惩罚因长翻译输出导致的延迟——这种延迟会阻碍用户理解及后续翻译进程。本文提出一种新型同声传译延迟评估指标——平均分词延迟(Average Token Delay, ATD),该指标聚焦于部分翻译的持续时间。我们通过基于耳语跨度(Ear-Voice Span, EVS)的模拟用户侧延迟分析,验证了其有效性。实验表明,在大多数条件下ATD与基线延迟指标中EVS的相关系数最高。