Human evaluation is the gold standard for multilingual NLP, but is often skipped in practice and substituted with automatic metrics, because it is notoriously complex and slow to set up with existing tools with substantial engineering and operational overhead. We introduce Pearmut, a lightweight yet feature-rich platform that makes end-to-end human evaluation as easy to run as automatic evaluation. Pearmut removes common entry barriers and provides support for evaluating multilingual tasks, with a particular focus on machine translation. The platform implements standard evaluation protocols, including DA, ESA, or MQM, but is also extensible to allow prototyping new protocols. It features document-level context, absolute and contrastive evaluation, attention checks, ESAAI pre-annotations and both static and active learning-based assignment strategies. Pearmut enables reliable human evaluation to become a practical, routine component of model development and diagnosis rather than an occasional effort.

翻译：人工评估是多语言自然语言处理领域的黄金标准，但在实践中常因现有工具设置复杂、耗时且需大量工程与运维投入而被跳过，转而采用自动评估指标替代。我们推出Pearmut这一轻量级但功能丰富的平台，使端到端人工评估变得与自动评估同样易于运行。该平台消除了常见的入门障碍，并为多语言任务（尤其聚焦机器翻译）的评估提供支持。平台实现了标准评估协议（包括DA、ESA和MQM），同时具备可扩展性以支持新协议的原型开发。其特色功能包括文档级上下文、绝对与对比评估、注意力检查、ESAAI预标注，以及基于静态和主动学习的分配策略。Pearmut使可靠的人工评估成为模型开发与诊断中实用、常规的组成部分，而非偶发性工作。