Language model (LM) probability is not a reliable quality estimator, as natural language is ambiguous. When multiple output options are valid, the model's probability distribution is spread across them, which can misleadingly indicate low output quality. This issue is caused by two reasons: (1) LMs' final output activation is softmax, which does not allow multiple correct options to receive high probabilities simultaneuously and (2) LMs' training data is single, one-hot encoded references, indicating that there is only one correct option at each output step. We propose training a module for Quality Estimation on top of pre-trained LMs to address these limitations. The module, called Sigmoid Head, is an extra unembedding head with sigmoid activation to tackle the first limitation. To tackle the second limitation, during the negative sampling process to train the Sigmoid Head, we use a heuristic to avoid selecting potentially alternative correct tokens. Our Sigmoid Head is computationally efficient during training and inference. The probability from Sigmoid Head is notably better quality signal compared to the original softmax head. As the Sigmoid Head does not rely on human-annotated quality data, it is more robust to out-of-domain settings compared to supervised QE.

翻译：语言模型（LM）概率并非可靠的质量估计指标，因为自然语言具有模糊性。当存在多个有效输出选项时，模型的概率分布会分散到这些选项上，从而可能误导性地表明输出质量较低。该问题由两个原因导致：（1）语言模型的最终输出激活函数为softmax，该函数不允许多个正确选项同时获得高概率；（2）语言模型的训练数据是单一的热编码参考序列，暗示每个输出步骤仅存在一个正确选项。为克服这些局限，我们提出在预训练语言模型之上训练质量估计模块。该模块称为Sigmoid头部，是一个采用sigmoid激活函数的额外解嵌入头部，用于解决第一个局限。针对第二个局限，在训练Sigmoid头部的负采样过程中，我们采用启发式方法避免选择可能存在的替代正确词元。我们的Sigmoid头部在训练和推理阶段均具有计算效率。与原始softmax头部相比，Sigmoid头部输出的概率是显著更优的质量信号。由于Sigmoid头部不依赖于人工标注的质量数据，其在领域外场景中比有监督质量估计方法具有更强的鲁棒性。