Representation learning, and interpreting learned representations, are key areas of focus in machine learning and neuroscience. Both fields generally use representations as a means to understand or improve a system's computations. In this work, however, we explore surprising dissociations between representation and computation that may pose challenges for such efforts. We create datasets in which we attempt to match the computational role that different features play, while manipulating other properties of the features or the data. We train various deep learning architectures to compute these multiple abstract features about their inputs. We find that their learned feature representations are systematically biased towards representing some features more strongly than others, depending upon extraneous properties such as feature complexity, the order in which features are learned, and the distribution of features over the inputs. For example, features that are simpler to compute or learned first tend to be represented more strongly and densely than features that are more complex or learned later, even if all features are learned equally well. We also explore how these biases are affected by architectures, optimizers, and training regimes (e.g., in transformers, features decoded earlier in the output sequence also tend to be represented more strongly). Our results help to characterize the inductive biases of gradient-based representation learning. These results also highlight a key challenge for interpretability $-$ or for comparing the representations of models and brains $-$ disentangling extraneous biases from the computationally important aspects of a system's internal representations.

翻译：表征学习与学习表征的解读是机器学习和神经科学领域的核心关注点。这两个领域通常将表征作为理解或改进系统计算能力的手段。然而，在本研究中，我们探究了表征与计算之间令人意外的分离现象，这可能对相关研究构成挑战。我们创建了数据集，试图在操控特征或数据其他属性的同时，匹配不同特征所扮演的计算角色。我们训练多种深度学习架构来计算其输入的多个抽象特征。研究发现，这些架构学习到的特征表征会系统性地偏向于更显著地表征某些特征，这取决于特征复杂度、学习顺序、特征在输入中的分布等外在属性。例如，即便所有特征都能被同等良好地学习，那些更易计算或更早学习的特征往往比更复杂或更晚学习的特征被更强烈、更密集地表征。我们还探究了这些偏差如何受架构、优化器和训练范式的影响（例如，在Transformer中，输出序列中较早解码的特征往往也会被更强地表征）。我们的研究有助于刻画基于梯度的表征学习的归纳偏好，同时揭示了可解释性研究（或模型与大脑表征比较研究）面临的关键挑战——如何从系统内部表征中剥离计算重要部分与外在偏差的纠缠。