Deep neural networks have achieved remarkable performance in retrieval-based dialogue systems, but they are shown to be ill calibrated. Though basic calibration methods like Monte Carlo Dropout and Ensemble can calibrate well, these methods are time-consuming in the training or inference stages. To tackle these challenges, we propose an efficient uncertainty calibration framework GPF-BERT for BERT-based conversational search, which employs a Gaussian Process layer and the focal loss on top of the BERT architecture to achieve a high-quality neural ranker. Extensive experiments are conducted to verify the effectiveness of our method. In comparison with basic calibration methods, GPF-BERT achieves the lowest empirical calibration error (ECE) in three in-domain datasets and the distributional shift tasks, while yielding the highest $R_{10}@1$ and MAP performance on most cases. In terms of time consumption, our GPF-BERT has an 8$\times$ speedup.

翻译：深度神经网络在检索式对话系统中取得了显著性能，但研究表明其存在校准不良的问题。尽管蒙特卡洛Dropout和集成等基础校准方法能够实现良好校准，但这些方法在训练或推理阶段耗时较长。针对这些挑战，我们提出了一种面向基于BERT的对话搜索的高效不确定性校准框架GPF-BERT，该框架在BERT架构之上引入高斯过程层与焦点损失函数，以实现高质量的神经排序器。通过大量实验验证了该方法的有效性。与基础校准方法相比，GPF-BERT在三个域内数据集与分布偏移任务上取得了最低的经验校准误差（ECE），同时在多数情况下实现了最高的$R_{10}@1$和MAP性能。在时间消耗方面，我们的GPF-BERT实现了8倍加速。