Learning features from data is one of the defining characteristics of deep learning, but our theoretical understanding of the role features play in deep learning is still rudimentary. To address this gap, we introduce a new tool, the interaction tensor, for empirically analyzing the interaction between data and model through features. With the interaction tensor, we make several key observations about how features are distributed in data and how models with different random seeds learn different features. Based on these observations, we propose a conceptual framework for feature learning. Under this framework, the expected accuracy for a single hypothesis and agreement for a pair of hypotheses can both be derived in closed-form. We demonstrate that the proposed framework can explain empirically observed phenomena, including the recently discovered Generalization Disagreement Equality (GDE) that allows for estimating the generalization error with only unlabeled data. Further, our theory also provides explicit construction of natural data distributions that break the GDE. Thus, we believe this work provides valuable new insight into our understanding of feature learning.

翻译：从数据中学习特征是深度学习最具标志性的特性之一，但我们对特征在深度学习中所起作用的理论理解仍十分初步。为弥补这一不足，我们引入了一种新工具——交互张量，用于通过特征实证分析数据与模型之间的交互作用。借助交互张量，我们获得了关于特征在数据中如何分布以及不同随机种子的模型如何学习不同特征的若干关键观察。基于这些观察，我们提出一个概念性的特征学习框架。在该框架下，单一假设的期望准确率以及一对假设的一致性均可通过闭式推导得出。我们证明该框架能够解释实证观察到的现象，包括最近发现的泛化差异等式（GDE），该等式允许仅使用未标注数据估计泛化误差。此外，我们的理论还提供了破坏GDE的自然数据分布的明确构造。因此，我们相信这项工作为理解特征学习提供了宝贵的新见解。