Running LLMs with extended reasoning on every problem is expensive, but determining which inputs actually require additional compute remains challenging. We investigate whether their own likelihood of success is recoverable from their internal representations before generation, and if this signal can guide more efficient inference. We train linear probes on pre-generation activations to predict policy-specific success on math and coding tasks, substantially outperforming surface features such as question length and TF-IDF. Using E2H-AMC, which provides both human and model performance on identical problems, we show that models encode a model-specific notion of difficulty that is distinct from human difficulty, and that this distinction increases with extended reasoning. Leveraging these probes, we demonstrate that routing queries across a pool of models can exceed the best-performing model whilst reducing inference cost by up to 70\% on MATH, showing that internal representations enable practical efficiency gains even when they diverge from human intuitions about difficulty. Our code is available at: https://github.com/KabakaWilliam/llms_know_difficulty

翻译：对每个问题运行具有扩展推理能力的大型语言模型（LLMs）成本高昂，但确定哪些输入实际需要额外计算资源仍然具有挑战性。我们研究模型在生成前能否从其内部表征中恢复自身成功的可能性，以及该信号是否能指导更高效的推理。我们在生成前激活上训练线性探针，以预测数学和编程任务中特定策略的成功率，其性能显著优于问题长度和TF-IDF等表层特征。通过使用E2H-AMC（该数据集在相同问题上同时提供人类和模型性能），我们发现模型编码了一种区别于人类困难程度的模型特定难度概念，且这种差异随扩展推理而增大。利用这些探针，我们证明在模型池中路由查询可以在MATH数据集上超越最佳单模型性能，同时将推理成本降低高达70%，这表明即使内部表征与人类对难度的直觉存在分歧，仍能实现实际效率提升。我们的代码发布于：https://github.com/KabakaWilliam/llms_know_difficulty