If you had an AI Whale-to-English translator, how could you validate whether or not it is working? Does one need to interact with the animals or rely on grounded observations such as temperature? We provide theoretical and proof-of-concept experimental evidence suggesting that interaction and even observations may not be necessary for sufficiently complex languages. One may be able to evaluate translators solely by their English outputs, offering potential advantages in terms of safety, ethics, and cost. This is an instance of machine translation quality evaluation (MTQE) without any reference translations available. A key challenge is identifying ``hallucinations,'' false translations which may appear fluent and plausible. We propose using segment-by-segment translation together with the classic NLP shuffle test to evaluate translators. The idea is to translate animal communication, turn by turn, and evaluate how often the resulting translations make more sense in order than permuted. Proof-of-concept experiments on data-scarce human languages and constructed languages demonstrate the potential utility of this evaluation methodology. These human-language experiments serve solely to validate our reference-free metric under data scarcity. It is found to correlate highly with a standard evaluation based on reference translations, which are available in our experiments. We also perform a theoretical analysis suggesting that interaction may not be necessary nor efficient in the early stages of learning to translate.

翻译：若有一款鲸语-英语AI翻译器，如何验证其是否有效？是否需要与动物互动或依赖温度等具身观测？我们提供的理论与概念验证实验证据表明，对于足够复杂的语言，互动乃至观测可能并非必需。仅通过其英语输出即可评估翻译器，这在安全性、伦理性和成本方面具有潜在优势。这是机器翻译质量评估（MTQE）中无任何参考译文的实例。关键挑战在于识别“幻觉”——那些看似流畅合理但实为错误的翻译。我们提出采用逐段翻译结合经典NLP乱序测试来评估翻译器。其核心思想是逐轮翻译动物通信，并评估所得翻译在原始顺序中比随机排列时更合理的频率。在数据稀缺的人类语言与构造语言上进行的原理验证实验证明了该评估方法的潜在效用。这些人类语言实验仅用于验证我们在数据稀缺条件下的无参考评估指标。实验发现该指标与基于参考译文（实验中可获取）的标准评估高度相关。我们还进行了理论分析，表明在翻译学习的早期阶段，互动可能既非必要亦非高效。