The training of large language models (LLMs) is expensive. In this paper, we study data-efficient approaches for pre-training LLMs, i.e., techniques that aim to optimize the Pareto frontier of model quality and training resource/data consumption. We seek to understand the tradeoffs associated with data selection routines based on (i) expensive-to-compute data-quality estimates, and (ii) maximization of coverage and diversity-based measures in the feature space. Our first technique, Ask-LLM, leverages the zero-shot reasoning capabilities of instruction-tuned LLMs to directly assess the quality of a training example. To target coverage, we propose Density sampling, which models the data distribution to select a diverse sample. In our comparison of 19 samplers, involving hundreds of evaluation tasks and pre-training runs, we find that Ask-LLM and Density are the best methods in their respective categories. Coverage sampling can recover the performance of the full data, while models trained on Ask-LLM data consistently outperform full-data training -- even when we reject 90% of the original dataset, while converging up to 70% faster.

翻译：大型语言模型（LLM）的训练成本高昂。本文研究了用于预训练LLM的数据高效方法，即旨在优化模型质量与训练资源/数据消耗的帕累托前沿的技术。我们试图理解基于（i）计算成本高昂的数据质量估计和（ii）特征空间中覆盖率和多样性度量最大化的数据选择例程相关的权衡。我们的第一种技术Ask-LLM，利用指令调优LLM的零样本推理能力来直接评估训练示例的质量。为了针对覆盖率，我们提出了密度采样，它通过建模数据分布来选择一个多样化的样本。在我们对19种采样器的比较中，涉及数百个评估任务和预训练运行，我们发现Ask-LLM和密度采样分别是各自类别中最好的方法。覆盖率采样能够恢复完整数据的性能，而使用Ask-LLM数据训练的模型始终优于全数据训练——即使我们拒绝了原始数据集的90%，同时收敛速度最多提高了70%。