Proper learning refers to the setting in which learners must emit predictors in the underlying hypothesis class $H$, and often leads to learners with simple algorithmic forms (e.g. empirical risk minimization (ERM), structural risk minimization (SRM)). The limitation of proper learning, however, is that there exist problems which can only be learned improperly, e.g. in multiclass classification. Thus, we ask: Under what assumptions on the hypothesis class or the information provided to the learner is a problem properly learnable? We first demonstrate that when the unlabeled data distribution is given, there always exists an optimal proper learner governed by distributional regularization, a randomized generalization of regularization. We refer to this setting as the distribution-fixed PAC model, and continue to evaluate the learner on its worst-case performance over all distributions. Our result holds for all metric loss functions and any finite learning problem (with no dependence on its size). Further, we demonstrate that sample complexities in the distribution-fixed PAC model can shrink by only a logarithmic factor from the classic PAC model, strongly refuting the role of unlabeled data in PAC learning (from a worst-case perspective). We complement this with impossibility results which obstruct any characterization of proper learnability in the realizable PAC model. First, we observe that there are problems whose proper learnability is logically undecidable, i.e., independent of the ZFC axioms. We then show that proper learnability is not a monotone property of the underlying hypothesis class, and that it is not a local property (in a precise sense). Our impossibility results all hold even for the fundamental setting of multiclass classification, and go through a reduction of EMX learning (Ben-David et al., 2019) to proper classification which may be of independent interest.

翻译：恰当学习指学习者必须输出底层假设类$H$中的预测器的设定，通常导致具有简单算法形式的学习器（例如经验风险最小化（ERM）、结构风险最小化（SRM））。然而，恰当学习的局限性在于存在只能被不恰当学习的问题，例如在多类分类中。因此，我们提出：在假设类或提供给学习者的信息满足何种假设时，问题才是恰当可学习的？我们首先证明，当给定未标记数据分布时，总存在一个由分布正则化（正则化的随机化推广）支配的最优恰当学习器。我们将此设定称为分布固定PAC模型，并继续评估学习器在所有分布上的最坏情况性能。我们的结果适用于所有度量损失函数和任何有限学习问题（与其规模无关）。进一步，我们证明分布固定PAC模型中的样本复杂度仅可能比经典PAC模型缩减对数因子，这从最坏情况视角强烈否定了未标记数据在PAC学习中的作用。我们通过不可能性结果对此进行补充，这些结果阻碍了在可实现PAC模型中对恰当可学习性的任何刻画。首先，我们观察到存在其恰当可学习性在逻辑上不可判定的问题（即独立于ZFC公理）。接着，我们证明恰当可学习性不是底层假设类的单调性质，也不是局部性质（在精确意义上）。我们的所有不可能性结果即使在多类分类这一基础设定中依然成立，并通过将EMX学习（Ben-David等人，2019年）归约为恰当分类来实现，这一归约可能具有独立研究价值。