Though Transformers have achieved promising results in many computer vision tasks, they tend to be over-confident in predictions, as the standard Dot Product Self-Attention (DPSA) can barely preserve distance for the unbounded input domain. In this work, we fill this gap by proposing a novel Lipschitz Regularized Transformer (LRFormer). Specifically, we present a new similarity function with the distance within Banach Space to ensure the Lipschitzness and also regularize the term by a contractive Lipschitz Bound. The proposed method is analyzed with a theoretical guarantee, providing a rigorous basis for its effectiveness and reliability. Extensive experiments conducted on standard vision benchmarks demonstrate that our method outperforms the state-of-the-art single forward pass approaches in prediction, calibration, and uncertainty estimation.

翻译：尽管Transformer在众多计算机视觉任务中取得了显著成果，但标准点积自注意力机制（DPSA）在无界输入域上难以保持距离特性，导致其预测容易过度自信。为此，本文提出了一种新颖的Lipschitz正则化Transformer（LRFormer）。具体而言，我们设计了一种基于巴拿赫空间内距离的新颖相似度函数以保障Lipschitz连续性，并通过收缩性Lipschitz边界对该项进行正则化。所提方法具有理论保证，为其有效性和可靠性提供了严谨依据。在标准视觉基准上的大量实验表明，本方法在预测准确性、校准质量和不确定性估计方面均优于现有的单次前向传播方法。