Deep neural networks trained in an end-to-end manner are proven to be efficient in a wide range of machine learning tasks. However, there is one drawback of end-to-end learning: The learned features and information are implicitly represented in neural network parameters, which cannot be used as regularities, concepts or knowledge to explicitly represent the data probability distribution. To resolve this issue, we propose in this paper a new machine learning theory, which defines in mathematics what are regularities. Briefly, regularities are concise representations of the non-random features, or 'non-randomness' in the data probability distribution. Combining this with information theory, we claim that regularities can also be regarded as a small amount of information encoding a large amount of information. Our theory is based on spiking functions. That is, if a function can react to, or spike on specific data samples more frequently than random noise inputs, we say that such a function discovers non-randomness from the data distribution. Also, we say that the discovered non-randomness is encoded into regularities if the function is simple enough. Our theory also discusses applying multiple spiking functions to the same data distribution. In this process, we claim that the 'best' regularities, or the optimal spiking functions, are those who can capture the largest amount of information from the data distribution, and then encode the captured information in the most concise way. Theorems and hypotheses are provided to describe in mathematics what are 'best' regularities and optimal spiking functions. Finally, we propose a machine learning approach, which can potentially obtain the optimal spiking functions regarding the given dataset in practice.

翻译：端到端训练的深度神经网络已被证明在广泛的机器学习任务中具有高效性。然而，端到端学习存在一个缺陷：学习到的特征和信息隐式地表示在神经网络参数中，无法作为规律性、概念或知识来显式地表征数据概率分布。为解决这一问题，本文提出一种新的机器学习理论，从数学上定义何为规律性。简而言之，规律性是数据概率分布中非随机特征或"非随机性"的简洁表示。结合信息论，我们认为规律性亦可视为以少量信息编码大量信息的载体。本理论基于脉冲函数构建。具体而言，若某函数对特定数据样本的响应（或脉冲发放）频率高于对随机噪声输入的响应，则称该函数从数据分布中发现了非随机性。同时，若该函数足够简单，我们称所发现的非随机性被编码为规律性。本理论还讨论了将多个脉冲函数应用于同一数据分布的过程。在此过程中，我们主张"最优"规律性（即最优脉冲函数）应能捕获数据分布中的最大信息量，并以最简洁的方式编码所捕获的信息。本文通过定理与假说从数学角度描述了何为"最优"规律性与最优脉冲函数。最后，我们提出一种机器学习方法，在实践中可针对给定数据集潜在获得最优脉冲函数。