The high cost of fine-tuning LLMs poses a significant economic barrier; pre-hoc performance prediction offers a critical solution to substantially reduce this expense. However, the theoretical limits of pre-hoc performance prediction remain unexplored. We formulate it as a stochastic estimation problem under information constraints, decomposing prediction risk into two components: an intrinsic limit (static data-model compatibility) and a reducible optimization variance. We prove that optimization variance admits a necessary lower bound on its decay rate, implying fundamental constraints on how quickly uncertainty dissipates, regardless of the predictor used. Based on these dynamics, we derive a budget-optimal probing principle and introduce a predictability phase diagram that organizes tasks into three distinct regimes: Static-Sufficient, Dynamic-Critical, and Noise-Dominant. Extensive experiments on synthetic and real-world benchmarks validate these theoretical regimes and demonstrate the efficiency of our probing strategy.

翻译：微调大型语言模型的高昂成本构成了显著的经济障碍；预优性能预测提供了一种关键解决方案，可大幅降低此开销。然而，预优性能预测的理论极限尚未被探索。我们将其构建为信息约束下的随机估计问题，将预测风险分解为两个组成部分：固有极限（静态数据-模型兼容性）和可缩减的优化方差。我们证明了优化方差在衰减速率上存在必要下界，这意味着无论使用何种预测器，不确定性消散的速度都受到基本约束。基于这些动态特性，我们推导出了预算最优的探测原则，并引入了可预测性相图，将任务划分为三个截然不同的区域：静态充分区、动态临界区和噪声主导区。在合成基准和真实世界基准上的大量实验验证了这些理论区域，并展示了我们探测策略的效率。