The rapid rise in popularity of large language models (LLMs) for translation calls for a thorough study of the reliability of their confidence in their own outputs. Unlike many generation tasks, translation errors and confidence levels can be useful at different levels of granularity (tokens, words, or spans). Unsupervised approaches based on internal signals like predicted probabilities can be misleading because they reflect certainty among alternatives rather than correctness. In addition, they require access to such internal signals. Here, we devise five verbalized methods of extracting an LLM's per-token confidence without those shortcomings and compare their reliability with that of the model's internal signals of certainty. We evaluate reliability using two forms of alignment: fine-grained error detection and calibration. For both, internal and verbalized methods perform similarly, although results vary by model. Interestingly, we find little to no correlation between internal and verbalized methods.

翻译：大型语言模型（LLM）在翻译领域的快速普及要求我们对其自身输出置信度的可靠性进行深入研究。与许多生成任务不同，翻译错误和置信度可以在不同粒度级别（标记、词语或片段）上发挥实际作用。基于预测概率等内部信号的无监督方法可能具有误导性，因为它们反映的是替代方案间的确定性而非正确性，且需要访问此类内部信号。本文设计了五种无需这些缺陷就能提取LLM逐标记置信度的口头化方法，并将其可靠性与模型内部确定性信号进行对比。我们通过两种对齐形式评估可靠性：细粒度错误检测和校准。对于两者，口头化方法与内部方法表现相近，尽管结果因模型而异。值得注意的是，我们发现口头化方法与内部方法之间几乎不存在相关性。