Progress in large language models is increasingly constrained by an evaluation bottleneck: benchmarks must be built and models run before iteration can begin. We investigate whether evaluation outcomes can be forecast before any experiments are conducted. Specifically, we study text-only performance prediction, where models estimate performance from task descriptions and experimental configurations alone, without access to dataset instances. To support systematic study, we curate PRECOG, a corpus of description-performance pairs spanning diverse tasks, domains, and metrics. We scrape task and configuration descriptions from arXiv, yielding 2,290 instances covering 1,519 papers, and construct a test split using papers published after the evaluated models' knowledge cutoff. Experiments show the task is challenging but feasible: reasoning models achieve a non-trivial forecasting skill reaching mean absolute error as low as 9.9 at high-confidence thresholds. Overall, our corpus and analyses offer an initial step toward open-ended anticipatory evaluation, supporting difficulty estimation and smarter resource allocation.

翻译：大型语言模型的进展日益受到评估瓶颈的制约：基准测试必须先行构建，模型必须运行，然后迭代才能开始。我们研究是否可以在任何实验进行之前预测评估结果。具体而言，我们研究纯文本性能预测，即模型仅从任务描述和实验配置中估计性能，而无需访问数据集实例。为了支持系统性研究，我们构建了PRECOG语料库，这是一个涵盖多样化任务、领域和指标的描述-性能对集合。我们从arXiv抓取任务和配置描述，获得了涵盖1,519篇论文的2,290个实例，并使用在被评估模型知识截止日期之后发表的论文构建了测试集。实验表明该任务具有挑战性但可行：推理模型在较高置信度阈值下实现了非平凡的预测能力，平均绝对误差可低至9.9。总体而言，我们的语料库和分析为开放式预期性评估迈出了第一步，支持难度估计和更智能的资源分配。