Results in interpretability suggest that large vision and language models learn implicit linear encodings when models are biased by in-context prompting. However, the existence of similar linear representations in more general adaptation regimes has not yet been demonstrated. In this work, we develop the concept of a task matrix, a linear transformation from a base to finetuned embedding state. We demonstrate that for vision and text models and ten different datasets, a base model augmented with a task matrix achieves results surpassing linear probes, sometimes approaching finetuned levels. Our results validate the existence of cross-layer linear encodings between pretrained and finetuned architectures. Moreover, we show that a data-based approximation for such encodings is both efficient and generalizable to multiple domains. We make our implementation publicly available.

翻译：可解释性研究结果表明，大型视觉和语言模型在上下文提示的偏置下会学习隐式的线性编码。然而，在更一般的适应机制中是否存在类似的线性表示尚未得到证实。在本研究中，我们提出了任务矩阵的概念，即从基础嵌入状态到微调嵌入状态的线性变换。我们证明，对于视觉和文本模型以及十个不同的数据集，配备任务矩阵的基础模型能够取得超越线性探针的结果，有时甚至接近微调后的性能水平。我们的结果验证了预训练与微调架构之间存在跨层线性编码。此外，我们表明基于数据的此类编码近似方法既高效又可推广至多个领域。我们已公开提供实现代码。