Large Language Models (LLMs) are demonstrating rapid improvements on complex reasoning benchmarks, particularly when allowed to utilize intermediate reasoning steps before converging on a final solution. However, current literature often overlooks the significant computational burden associated with generating long reasoning sequences. For industrial applications, model selection depends not only on raw accuracy but also on resource constraints and inference costs. In this work, we conduct a test-time-compute aware evaluation of both contemporary and older open-source LLMs, mapping their Pareto frontiers across math- and reasoning-intensive benchmarks. Our findings identify the Mixture of Experts (MoE) architecture as a strong candidate to balance performance and efficiency in our evaluation setting. Furthermore, we trace the trajectory of Pareto efficiency over time to derive an emergent trend regarding accuracy gain per unit of compute. Finally, we demonstrate that there is a saturation point for inference-time compute. Beyond a certain threshold, accuracy gains diminish, indicating that while extended reasoning capabilities are beneficial, they cannot overcome intrinsic model limitations regarding specific complexities.

翻译：大语言模型（LLMs）在复杂推理基准测试中正展现出快速进步，尤其是在允许其利用中间推理步骤再收敛至最终解的情况下。然而，当前文献往往忽视了生成长推理序列所带来的显著计算负担。对于工业应用而言，模型选择不仅取决于原始精度，还受到资源约束和推理成本的影响。在本工作中，我们对当代及早期的开源LLMs进行了考虑测试时计算成本的评估，绘制了它们在数学与推理密集型基准测试上的帕累托前沿。我们的研究结果表明，混合专家（MoE）架构在我们的评估设置中是平衡性能与效率的有力候选方案。此外，我们追踪了帕累托效率随时间变化的轨迹，以推导出关于单位计算量所获精度增益的涌现趋势。最后，我们证明了推理时计算存在一个饱和点。超过特定阈值后，精度增益将减弱，这表明尽管扩展推理能力是有益的，但其无法克服模型在特定复杂性方面固有的局限性。