Large language model (LLM) queries are predominantly processed by frontier models in centralized cloud infrastructure. Rapidly growing demand strains this paradigm, and cloud providers struggle to scale infrastructure at pace. Two advances enable us to rethink this paradigm: small LMs (<=20B active parameters) now achieve competitive performance to frontier models on many tasks, and local accelerators (e.g., Apple M4 Max) run these models at interactive latencies. This raises the question: can local inference viably redistribute demand from centralized infrastructure? Answering this requires measuring whether local LMs can accurately answer real-world queries and whether they can do so efficiently enough to be practical on power-constrained devices (i.e., laptops). We propose intelligence per watt (IPW), task accuracy divided by unit of power, as a metric for assessing capability and efficiency of local inference across model-accelerator pairs. We conduct a large-scale empirical study across 20+ state-of-the-art local LMs, 8 accelerators, and a representative subset of LLM traffic: 1M real-world single-turn chat and reasoning queries. For each query, we measure accuracy, energy, latency, and power. Our analysis reveals $3$ findings. First, local LMs can accurately answer 88.7% of single-turn chat and reasoning queries with accuracy varying by domain. Second, from 2023-2025, IPW improved 5.3x and local query coverage rose from 23.2% to 71.3%. Third, local accelerators achieve at least 1.4x lower IPW than cloud accelerators running identical models, revealing significant headroom for optimization. These findings demonstrate that local inference can meaningfully redistribute demand from centralized infrastructure, with IPW serving as the critical metric for tracking this transition. We release our IPW profiling harness here: https://github.com/HazyResearch/intelligence-per-watt.

翻译：大型语言模型（LLM）的查询主要由前沿模型在集中式云基础设施中处理。需求的快速增长使该范式面临压力，云服务提供商难以同步扩展基础设施。两项进展使我们能够重新思考这一范式：小型语言模型（≤200亿活跃参数）如今在许多任务上已达到与前沿模型相竞争的性能，而本地加速器（例如 Apple M4 Max）能够以交互式延迟运行这些模型。这引发了一个问题：本地推理能否切实可行地将需求从集中式基础设施中重新分配？回答此问题需要衡量本地语言模型是否能准确回答现实世界查询，以及其效率是否足以在功耗受限的设备（如笔记本电脑）上实用。我们提出智能每瓦（IPW），即任务准确率除以单位功耗，作为评估跨模型-加速器对的本地推理能力与效率的指标。我们在超过20个最先进的本地语言模型、8种加速器以及一个具有代表性的大型语言模型流量子集（100万个现实世界单轮对话和推理查询）上进行了大规模实证研究。针对每个查询，我们测量了准确率、能耗、延迟和功耗。我们的分析揭示了三点发现。首先，本地语言模型能够准确回答88.7%的单轮对话和推理查询，其准确率因领域而异。其次，从2023年到2025年，IPW提升了5.3倍，本地查询覆盖率从23.2%上升至71.3%。第三，本地加速器相比运行相同模型的云加速器，其IPW至少降低了1.4倍，显示出显著的优化空间。这些发现表明，本地推理能够有意义地重新分配来自集中式基础设施的需求，而IPW是追踪这一转变的关键指标。我们在此发布我们的IPW分析工具：https://github.com/HazyResearch/intelligence-per-watt。