A recent body of work has demonstrated that Transformer embeddings can be linearly decomposed into well-defined sums of factors, that can in turn be related to specific network inputs or components. There is however still a dearth of work studying whether these mathematical reformulations are empirically meaningful. In the present work, we study representations from machine-translation decoders using two of such embedding decomposition methods. Our results indicate that, while decomposition-derived indicators effectively correlate with model performance, variation across different runs suggests a more nuanced take on this question. The high variability of our measurements indicate that geometry reflects model-specific characteristics more than it does sentence-specific computations, and that similar training conditions do not guarantee similar vector spaces.

翻译：近期研究表明，Transformer嵌入可线性分解为定义明确的因子之和，这些因子进而与特定网络输入或组件相关联。然而，目前仍缺乏关于这些数学重构是否具有实证意义的研究。本文采用两种嵌入分解方法，对机器翻译解码器的表示进行了研究。结果表明：尽管基于分解的指标与模型性能存在有效关联，但不同运行间的差异性提示我们需对此问题持更审慎态度。测量值的高度变异性表明，几何结构更多反映的是模型特异性特征而非句子特异性计算过程，且相似的训练条件并不能保证相似的向量空间。