Sequence labeling is a core task in text understanding for IE/IR systems. Text generation models have increasingly become the go-to solution for such tasks (e.g., entity extraction and dialog slot filling). While most research has focused on the labeling accuracy, a key aspect -- of vital practical importance -- has slipped through the cracks: understanding model confidence. More specifically, we lack a principled understanding of how to reliably gauge the confidence of a model in its predictions for each labeled span. This paper aims to provide some empirical insights on estimating model confidence for generative sequence labeling. Most notably, we find that simply using the decoder's output probabilities \textbf{is not} the best in realizing well-calibrated confidence estimates. As verified over six public datasets of different tasks, we show that our proposed approach -- which leverages statistics from top-$k$ predictions by a beam search -- significantly reduces calibration errors of the predictions of a generative sequence labeling model.

翻译：序列标注是信息检索/信息抽取系统中文本理解的核心任务。文本生成模型日益成为此类任务（例如实体抽取和对话槽填充）的首选解决方案。尽管大多数研究聚焦于标注准确率，但一个具有重要实际意义的关键方面却被忽视：理解模型置信度。具体而言，我们缺乏如何可靠评估模型对每个标注跨度预测置信度的原理性理解。本文旨在为生成式序列标注的置信度估计提供一些经验性见解。最为显著的是，我们发现单纯使用解码器输出概率**并非**实现良好校准置信度估计的最佳方案。通过六个不同任务公开数据集的验证，我们证明所提出的方法——利用波束搜索中前k个预测的统计量——能够显著降低生成式序列标注模型预测的校准误差。