It is often advantageous to train models on a subset of the available train examples, because the examples are of variable quality or because one would like to train with fewer examples, without sacrificing performance. We present Gradient Information Optimization (GIO), a scalable, task-agnostic approach to this data selection problem that requires only a small set of (unlabeled) examples representing a target distribution. GIO begins from a natural, information-theoretic objective that is intractable in practice. Our contribution is in showing that it can be made highly scalable through a simple relaxation of the objective and a highly efficient implementation. In experiments with machine translation, spelling correction, and image recognition, we show that GIO delivers outstanding results with very small train sets. These findings are robust to different representation models and hyperparameters for GIO itself. GIO is task- and domain-agnostic and can be applied out-of-the-box to new datasets and domains.

翻译：摘要：训练模型时，通常更优的做法是从可用训练样本中选取一个子集，这是因为样本质量参差不齐，或在希望减少训练样本数量而不牺牲模型性能的场景。本文提出梯度信息优化（GIO）方法，这是一种可扩展、任务无关的数据选择方案，仅需少量代表目标分布的（无标签）样本。GIO基于一项自然但实际中难以处理的信息论目标函数。我们的贡献在于证明，通过对该目标函数进行简单松弛并辅以高度高效的实现，可以使其具备极强的可扩展性。在机器翻译、拼写纠正和图像识别实验中，我们展示出GIO在极小训练集上取得了优异的结果。这些发现对不同的表征模型和GIO自身的超参数均具有鲁棒性。GIO具有任务与领域无关性，可直接应用于新的数据集和领域。