Open-source Lightreason claims 6-11x Qwen inference speedups on one RTX 4090

An open-source project called Lightreason reached the Hacker News front page on 13 November 2026 with a striking claim: Qwen language models generating tokens six to eleven times faster on a single consumer GPU, an RTX 4090. The repository, published under the GitHub account me-farza, presents itself as a local inference workbench with an interactive Studio for side-by-side benchmarking, and it ships source code, pinned download manifests and benchmark evidence — but no model weights.

The measurements, runner by runner

According to the repository, the experiments use original BF16 weights for two models: Qwen2.5-3B-Instruct at 3.09 billion parameters, and the text model of Qwen3.6-35B-A3B at 34.66 billion.

For the 3B model, the main Lightreason Exact runner lifted throughput from 63.0 to 139.6 tokens per second across six prompts. The author reports that it matched all 30 Hugging Face timing runs used for comparison and produced identical Python, JavaScript and SQL responses in nine out of nine new test pairs.

The top of the 6-11x range comes from a separate draft-assisted runner built on AMD's PARD2, which reached 525.2 tokens per second on a matching weather response, up from 58.3, and 724.4 tokens per second on the best matching 128-token Fibonacci slice, up from 63.6. There is a catch: the repository labels these as archived research measurements, and reproducing them requires a lab adapter that is not part of the public release. The target-only 3B results, by contrast, need only the public Qwen download.

The 35B model is the harder case. It went from 7.93 to 51.08 tokens per second on a capped Python prompt, a 6.44x comparison run with the same input across sessions. Outputs are not always identical at this scale: the run first diverged from the baseline Scheduled runner at token 116, and the packed configuration matched all 20 test pairs against a fixed eight-position control rather than the plain baseline. A draft-assisted 3B runner matched Hugging Face outputs in only 12 of 36 held-out comparisons, although it ran 1.062x faster than the previous draft-assisted version with unchanged output.

How the speedups are achieved

The 3B model fits in GPU memory, so the gains there come from systems work: capturing prepared GPU work once and replaying it, an application of NVIDIA's CUDA Graphs; a tighter attention cache; combined BF16 projections; and pre-filtering of vocabulary candidates, with the survivors rescored by the model's original output head. The original BF16 target still makes every final token decision, which is why outputs can remain identical.

The 35B model's text weights occupy 64.56 GiB, far too large for one consumer card, so the runner keeps most expert weights in system RAM and moves only the experts the model actually routes to. That hybrid CPU/GPU executor and the parallel draft come from FlashML's FreeToken and Inco AI's DFlash2. Lightreason adds prefill that fetches only selected rows, AVX2 reuse when expert routes repeat, native CUDA unpacking, packed host banks across 34 layers, and a lossless GPU replica tier, restoring the exact original BF16 values before expert computation. When a draft proposes several tokens, the target model verifies them together and reuses weights across those checks.

The project is explicit that these are assembled techniques rather than novel research, crediting CUDA Graphs, PARD2, DFlash2 and FreeToken, and its experiment history records which attempts slowed things down or changed the output.

Running it yourself

The quickstart targets Linux on an RTX 4090 with a CUDA 12.8-compatible driver, Python 3.12, PyTorch 2.9.1 and Transformers 4.57.1. The 3B download is about 6.2 GB, with another 2 GiB of free space required during the process; the 35B setup needs a separate environment with considerably more system RAM. A sparse-checkout command fetches only the 3B Studio and its results. The Studio serves a local web page that runs a chosen prompt through the standard engine and then Lightreason Exact on the same GPU, streaming both responses and reporting timings plus whether the token sequences match. Headless CLI runs emit JSON with preparation time, generation time, token IDs and stop reason. A porting guide covers adapting the approach to other hardware, including a bundled skill that coding agents can consume, and the repository reminds users to read the Qwen license before downloading the models.

Why it matters

A 35B-class mixture-of-experts model generating roughly 50 tokens per second on a single consumer GPU, and a 3B model at around 140 tokens per second with claimed output parity, is a genuinely practical result for local LLM tooling. It shows that large gains can still come from engineering — graph capture, cache layout, expert offloading, batched draft verification — stacked on existing open components, without quantizing away the original BF16 weights. The transparency is also unusual for a self-published speedup claim: pinned versions, published benchmark evidence and a log of failed experiments make the results checkable. The caveats matter just as much. Every figure is the author's own single-machine measurement, the fastest numbers depend on an adapter that is not public, and some fast paths trade exact output parity for speed. Independent replication on other hardware will determine whether Lightreason becomes a template for local inference or stays a well-documented one-off.