Large transformers are powerful architectures used for self-supervised data analysis across various data types, including protein sequences, images, and text. In these models, the semantic structure of the dataset emerges from a sequence of transformations between one representation and the next. We characterize the geometric and statistical properties of these representations and how they change as we move through the layers. By analyzing the intrinsic dimension (ID) and neighbor composition, we find that the representations evolve similarly in transformers trained on protein language tasks and image reconstruction tasks. In the first layers, the data manifold expands, becoming high-dimensional, and then contracts significantly in the intermediate layers. In the last part of the model, the ID remains approximately constant or forms a second shallow peak. We show that the semantic information of the dataset is better expressed at the end of the first peak, and this phenomenon can be observed across many models trained on diverse datasets. Based on our findings, we point out an explicit strategy to identify, without supervision, the layers that maximize semantic content: representations at intermediate layers corresponding to a relative minimum of the ID profile are more suitable for downstream learning tasks.

翻译：大型Transformer是一种强大的架构，用于跨多种数据类型（包括蛋白质序列、图像和文本）进行自监督数据分析。在这些模型中，数据集的语义结构通过表示之间的连续变换序列而涌现。我们表征了这些表示的几何与统计特性，以及它们随层数增加而变化的规律。通过分析内在维度（ID）和邻居构成，我们发现Transformer在蛋白质语言任务和图像重建任务上的表示演化具有相似性。在初始层中，数据流形膨胀并变得高维，随后在中间层显著收缩。在模型最后部分，内在维度大致保持恒定或形成第二个浅峰。我们表明数据集语义信息在第一个峰值末尾处表达更佳，且这一现象在多个基于不同数据集训练的模型中均可观察到。基于此发现，我们提出一种无需监督即可识别最大化语义内容层的显式策略：对应内在维度轮廓相对最小值的中间层表示，更适合下游学习任务。