We investigate the predictability of large language model (LLM) capabilities: given records of past experiments using different model families, numbers of parameters, tasks, and numbers of in-context examples, can we accurately predict LLM performance on new experiment configurations? Answering this question has practical implications for LLM users (e.g., deciding which models to try), developers (e.g., prioritizing evaluation on representative tasks), and the research community (e.g., identifying hard-to-predict capabilities that warrant further investigation). We study the performance prediction problem on experiment records from BIG-bench. On a random train-test split, an MLP-based predictor achieves an $R^2$ score greater than 95%, indicating the presence of learnable patterns within the experiment records. We then formulate the problem of searching for "small-bench," an informative subset of BIG-bench tasks from which the performance on the full set can be maximally recovered. We find a subset as informative as BIG-bench Hard for evaluating new model families, while being $3\times$ smaller. Additionally, we find competitive subsets by clustering task representations learned by our MLP-based predictor and selecting tasks close to cluster centroids, highlighting the importance of task diversity in constructing "small-bench."

翻译：我们研究了大型语言模型（LLM）能力的可预测性问题：给定使用不同模型系列、参数量、任务以及上下文示例数量的历史实验记录，我们能否准确预测LLM在新实验配置下的性能？回答这一问题对LLM用户（例如决定尝试哪些模型）、开发者（例如优先评估代表性任务）以及研究社区（例如识别难以预测且需进一步研究的能力）具有实际意义。我们针对BIG-bench的实验记录研究了性能预测问题。在随机划分的训练-测试集上，基于MLP的预测器达到了超过95%的$R^2$分数，表明实验记录中存在可学习的模式。接着，我们提出了搜索"小基准（small-bench）"的问题，即寻找BIG-bench任务中一个信息丰富的子集，使得从该子集上的性能能够最大程度地恢复完整集上的性能。我们找到了一个与BIG-bench Hard同样信息丰富、但规模缩小3倍的子集，用于评估新模型系列。此外，我们通过对MLP预测器学习到的任务表示进行聚类，并选择靠近聚类中心的任务，找到了具有竞争力的子集，这凸显了任务多样性在构建"小基准"中的重要性。