Stanford team proposes intelligence-per-watt metric for local AI efficiency

A Stanford-led research team has proposed a single number for judging whether an AI workload belongs on a laptop instead of in a datacenter: intelligence per watt, or IPW, defined as task accuracy per unit of power consumed. The paper, posted on arXiv after surfacing on Hacker News's front page, argues that local inference has improved enough to take a meaningful slice of demand away from centralized cloud infrastructure — and that IPW is the metric to track that transition with. The author list includes John Hennessy and Christopher Ré.

The premise the paper starts from

According to the abstract, most large language model queries are currently processed by frontier models running in centralized cloud infrastructure, and demand is growing faster than providers can scale that capacity. Two recent shifts create an opening to rethink the setup: small local models — the paper focuses on those with 20B or fewer active parameters — now achieve competitive performance with frontier models on many tasks, and local accelerators such as Apple's M4 Max can host these models at interactive latencies.

That raises the study's central question: can local inference viably redistribute demand from centralized infrastructure? Answering it requires measuring two things at once — whether local models answer real queries accurately, and whether they do so efficiently on power-constrained devices such as laptops. IPW is the authors' answer: one unified metric covering both capability and efficiency across model-and-accelerator combinations.

How the evaluation was run

The study covers more than 20 state-of-the-art local LMs, 8 hardware accelerators spanning local and cloud, and 1 million real-world single-turn chat and reasoning queries. For each query, the team measured accuracy — expressed as the local model's win rate against frontier models — along with energy, latency and power.

What the study found

Three results stand out. Local LMs successfully answered 88.7% of the queries, with accuracy varying by domain. A longitudinal analysis covering 2023 to 2025 shows IPW improved 5.3x, which the authors attribute to a combination of algorithmic and accelerator advances; over the same period, locally serviceable query coverage rose from 23.2% to 71.3%. And when identical models were run on both classes of hardware, local accelerators achieved at least 1.4x lower IPW than cloud accelerators — a gap the authors present as significant headroom for local accelerator optimization, since laptop-class chips still trail datacenter silicon on efficiency per watt.

Some caveats are worth holding onto. The work is a preprint, first posted in November 2025 and revised repeatedly since, most recently in September 2026. And because accuracy is defined as a win rate against frontier models, local capability is measured against a moving target.

Why it matters

For anyone running local AI under power constraints — laptops, edge devices, anything on a battery or a tight thermal budget — IPW offers a common yardstick where public discussion usually stops at model leaderboards and raw tokens-per-second. It collapses the two questions that actually decide a deployment into one number: is the model accurate enough for the task, and can the hardware sustain it within its power envelope.

The findings also sketch a routing argument. If local models handled 88.7% of real-world queries successfully in this evaluation, a substantial subset of traffic has no strict need for a frontier-cloud round trip — which is exactly the demand redistribution the paper anticipates, along with the energy and scaling pressure that come with it. The 1.4x efficiency gap, meanwhile, doubles as a benchmark for chipmakers: it quantifies how far local accelerators have to go before the local-first case gets easier to make.