Transformer models have achieved profound success in prediction tasks in a wide range of applications in natural language processing, speech recognition and computer vision. Extending Transformer's success to safety-critical domains requires calibrated uncertainty estimation which remains under-explored. To address this, we propose Sparse Gaussian Process attention (SGPA), which performs Bayesian inference directly in the output space of multi-head attention blocks (MHAs) in transformer to calibrate its uncertainty. It replaces the scaled dot-product operation with a valid symmetric kernel and uses sparse Gaussian processes (SGP) techniques to approximate the posterior processes of MHA outputs. Empirically, on a suite of prediction tasks on text, images and graphs, SGPA-based Transformers achieve competitive predictive accuracy, while noticeably improving both in-distribution calibration and out-of-distribution robustness and detection.

翻译：Transformer模型在自然语言处理、语音识别和计算机视觉等广泛应用的预测任务中取得了显著成功。将Transformer的成功扩展到安全关键领域需要校准的不确定性估计，而这一方向仍待深入探索。为此，我们提出稀疏高斯过程注意力（SGPA），该方法直接在Transformer多头注意力块（MHA）的输出空间中进行贝叶斯推理，以校准其不确定性。它采用有效对称核替代缩放点积运算，并利用稀疏高斯过程（SGP）技术近似MHA输出的后验过程。在文本、图像和图数据的系列预测任务中，基于SGPA的Transformer在保持竞争性预测精度的同时，显著提升了分布内校准性能以及分布外鲁棒性与检测能力。