Can we learn more from data than existed in the generating process itself? Can new and useful information be constructed from merely applying deterministic transformations to existing data? Can the learnable content in data be evaluated without considering a downstream task? On these questions, Shannon information and Kolmogorov complexity come up nearly empty-handed, in part because they assume observers with unlimited computational capacity and fail to target the useful information content. In this work, we identify and exemplify three seeming paradoxes in information theory: (1) information cannot be increased by deterministic transformations; (2) information is independent of the order of data; (3) likelihood modeling is merely distribution matching. To shed light on the tension between these results and modern practice, and to quantify the value of data, we introduce epiplexity, a formalization of information capturing what computationally bounded observers can learn from data. Epiplexity captures the structural content in data while excluding time-bounded entropy, the random unpredictable content exemplified by pseudorandom number generators and chaotic dynamical systems. With these concepts, we demonstrate how information can be created with computation, how it depends on the ordering of the data, and how likelihood modeling can produce more complex programs than present in the data generating process itself. We also present practical procedures to estimate epiplexity which we show capture differences across data sources, track with downstream performance, and highlight dataset interventions that improve out-of-distribution generalization. In contrast to principles of model selection, epiplexity provides a theoretical foundation for data selection, guiding how to select, generate, or transform data for learning systems.

翻译：我们能否从数据中学到比生成过程本身更多的信息？仅通过对现有数据应用确定性变换，能否构造出新的有用信息？在不考虑下游任务的情况下，能否评估数据中的可学习内容？对于这些问题，香农信息论与柯尔莫哥洛夫复杂性几乎无法给出答案，部分原因在于它们假设观察者具有无限计算能力，且未能针对有用信息内容进行建模。本研究识别并例证了信息论中三个看似矛盾的现象：(1) 信息无法通过确定性变换增加；(2) 信息与数据顺序无关；(3) 似然建模仅是分布匹配。为阐明这些结论与现代实践之间的张力，并量化数据的价值，我们提出了易学性这一概念——一种形式化的信息度量，用于刻画计算受限观察者能从数据中学习的内容。易学性捕捉数据中的结构信息，同时排除时间有界熵（即伪随机数生成器和混沌动力系统中典型的随机不可预测内容）。基于这些概念，我们论证了信息如何通过计算被创造，如何依赖于数据排序，以及似然建模如何产生比数据生成过程本身更复杂的程序。我们还提出了估计易学性的实用方法，实验表明该方法能有效区分不同数据源，与下游任务性能相关，并能识别提升分布外泛化能力的数据干预策略。与模型选择原则不同，易学性为数据选择提供了理论基础，指导如何为学习系统选择、生成或转换数据。