The challenge of low-latency speech translation has recently draw significant interest in the research community as shown by several publications and shared tasks. Therefore, it is essential to evaluate these different approaches in realistic scenarios. However, currently only specific aspects of the systems are evaluated and often it is not possible to compare different approaches. In this work, we propose the first framework to perform and evaluate the various aspects of low-latency speech translation under realistic conditions. The evaluation is carried out in an end-to-end fashion. This includes the segmentation of the audio as well as the run-time of the different components. Secondly, we compare different approaches to low-latency speech translation using this framework. We evaluate models with the option to revise the output as well as methods with fixed output. Furthermore, we directly compare state-of-the-art cascaded as well as end-to-end systems. Finally, the framework allows to automatically evaluate the translation quality as well as latency and also provides a web interface to show the low-latency model outputs to the user.

翻译：低延迟语音翻译的挑战近期因多项发表和共享任务引发了研究界的广泛关注。因此，在真实场景中对这些不同方法进行评估至关重要。然而，当前仅对系统的特定方面进行评估，且通常无法比较不同方法。本研究提出了首个在真实条件下执行并评估低延迟语音翻译多方面的框架。该评估以端到端方式进行，涵盖音频分割以及不同组件的运行时间。其次，我们利用该框架比较了低延迟语音翻译的不同方法。我们评估了具备输出修正选项的模型以及固定输出方法。此外，我们直接比较了当前最先进的级联系统与端到端系统。最后，该框架能够自动评估翻译质量与延迟，并提供网页界面以向用户呈现低延迟模型输出。