Two main concepts studied in machine learning theory are generalization gap (difference between train and test error) and excess risk (difference between test error and the minimum possible error). While information-theoretic tools have been used extensively to study the generalization gap of learning algorithms, the information-theoretic nature of excess risk has not yet been fully investigated. In this paper, some steps are taken toward this goal. We consider the frequentist problem of minimax excess risk as a zero-sum game between the algorithm designer and the world. Then, we argue that it is desirable to modify this game in a way that the order of play can be swapped. We then prove that, under some regularity conditions, if the world and designer can play randomly the duality gap is zero and the order of play can be changed. In this case, a Bayesian problem surfaces in the dual representation. This makes it possible to utilize recent information-theoretic results on minimum excess risk in Bayesian learning to provide bounds on the minimax excess risk. We demonstrate the applicability of the results by providing information theoretic insight on two important classes of problems: classification when the hypothesis space has finite VC-dimension, and regularized least squares.

翻译：机器学习理论中研究的两个核心概念是泛化差距（训练误差与测试误差之差）和超额风险（测试误差与最小可能误差之差）。尽管信息论工具已被广泛用于研究学习算法的泛化差距，但超额风险的信息论本质尚未得到充分探索。本文朝此目标迈出了若干步骤。我们将极小极大超额风险的频率派问题视为算法设计者与“世界”之间的零和博弈。进而论证，以改变博弈顺序的方式修正该博弈是可取的。随后证明，在若干正则性条件下，若“世界”与设计者均可随机行动，则对偶间隙为零且博弈顺序可被交换。此时，对偶表示中会浮现出一个贝叶斯问题。这使得我们能够利用贝叶斯学习中关于最小超额风险的最新信息论结果，为极小极大超额风险提供上界。通过为两类重要问题提供信息论洞见，我们展示了该方法的适用性：当假设空间具有有限VC维时的分类问题，以及正则化最小二乘问题。