Vision transformers (ViTs) - especially feature foundation models like DINOv2 - learn rich representations useful for many downstream tasks. However, architectural choices (such as positional encoding) can lead to these models displaying positional biases and artefacts independent of semantic content. This makes zero-shot adaption difficult in fields like material science, where images are often cross-sections of homogeneous microstructure (i.e. having no preferred direction). In this work, we investigate the positional bias in ViTs via linear probing, finding it present across a range of objectives and positional encodings, and subsequently reduce it by finetuning models to use ALiBi relative positional encoding. We demonstrate that these models retain desirable general semantics and their unbiased features can be used successfully in trainable segmentation of complex microscopy images.

翻译：视觉Transformer（ViT）——尤其是像DINOv2这样的特征基础模型——能够学习对多种下游任务有用的丰富表征。然而，架构选择（例如位置编码）可能导致这些模型表现出与语义内容无关的位置偏差和伪影。这使得在材料科学等领域进行零样本适配变得困难，因为这些领域的图像通常是均匀微观结构的截面（即没有优先方向）。在本研究中，我们通过线性探测探究了ViT中的位置偏差，发现该偏差存在于多种训练目标和位置编码方案中，随后通过微调模型使用ALiBi相对位置编码来降低这种偏差。我们证明这些模型保留了理想的通用语义特性，其无偏差特征可成功应用于复杂显微图像的可训练分割任务。