At the core of the Transformer, the softmax normalizes the attention matrix to be right stochastic. Previous research has shown that this often de-stabilizes training and that enforcing the attention matrix to be doubly stochastic (through Sinkhorn's algorithm) consistently improves performance across different tasks, domains and Transformer flavors. However, Sinkhorn's algorithm is iterative, approximative, non-parametric and thus inflexible w.r.t. the obtained doubly stochastic matrix (DSM). Recently, it has been proven that DSMs can be obtained with a parametric quantum circuit, yielding a novel quantum inductive bias for DSMs with no known classical analogue. Motivated by this, we demonstrate the feasibility of a hybrid classical-quantum doubly stochastic Transformer (QDSFormer) that replaces the softmax in the self-attention layer with a variational quantum circuit. We study the expressive power of the circuit and find that it yields more diverse DSMs that better preserve information than classical operators. Across multiple small-scale object recognition tasks, we find that our QDSFormer consistently surpasses both a standard ViT and other doubly stochastic Transformers. Beyond the Sinkformer, this comparison includes a novel quantum-inspired doubly stochastic Transformer (based on QR decomposition) that can be of independent interest. Our QDSFormer also shows improved training stability and lower performance variation suggesting that it may mitigate the notoriously unstable training of ViTs on small-scale data.

翻译：在Transformer的核心机制中，softmax函数通过归一化使注意力矩阵成为右随机矩阵。先前研究表明，这种处理常导致训练不稳定，而通过Sinkhorn算法强制注意力矩阵满足双随机性，能在不同任务、领域和Transformer变体中持续提升性能。然而，Sinkhorn算法具有迭代性、近似性和非参数性，导致其对生成的双随机矩阵缺乏灵活性。近期研究证明，双随机矩阵可通过参数化量子电路生成，这为双随机矩阵引入了一种无经典类比的新型量子归纳偏置。受此启发，我们提出了一种混合经典-量子双随机Transformer（QDSFormer），其用变分量子电路替代自注意力层中的softmax函数。我们研究了该电路的表达能力，发现其能生成比经典算子更多样化的双随机矩阵，并更好地保留信息。在多个小规模物体识别任务中，QDSFormer持续超越标准ViT及其他双随机Transformer变体。除Sinkformer外，本研究还包含一种基于QR分解的新型量子启发双随机Transformer（可能具有独立研究价值）。QDSFormer同时表现出更高的训练稳定性与更低的性能波动，表明其可能缓解ViT在小规模数据上众所周知的训练不稳定问题。