WebLLM runs LLMs entirely in the browser with WebGPU acceleration

WebLLM, an open-source inference engine from the MLC AI project, is drawing attention after surfacing on the front page of Hacker News. The library runs large language models entirely inside a web browser, with no backend involved, and leans on WebGPU to push the heavy computation onto the user's own GPU.

How the engine works

According to the project's GitHub repository, everything executes client-side: there is no server component, and hardware acceleration comes from WebGPU, the browser API that exposes modern GPU compute. WebLLM is presented as a companion to MLC LLM, a broader effort to make model deployment portable across different hardware targets. Models ship in MLC format, and the repository notes that structured JSON generation is handled by a WebAssembly component of the model library for better performance.

An OpenAI-compatible interface

A notable design choice is API compatibility. WebLLM exposes the OpenAI chat completion API, so code written against OpenAI's endpoints can be pointed at local open-source models instead. The repository lists streaming, JSON mode, logit-level control and seeding among the supported features, while function calling is still marked as work in progress. One behavioural difference: the familiar model parameter is ignored, and developers instead select a model when creating the engine or by calling its reload method.

Model lineup

The engine natively supports several model families, including Llama 3 and Llama 2, the Phi 3 series and its smaller predecessors, Gemma-2B, a set of Mistral-7B variants, and Qwen2 at 0.5B, 1.5B and 7B parameter sizes. A code example in the documentation uses a quantised Llama 3.1 8B instruct model. Developers needing something else can compile custom models into MLC format or request additions through the project's issue tracker.

Integration and caching

WebLLM is distributed as a package installable through npm, yarn or pnpm, and it can also be imported directly from a CDN, which the repository says is usable immediately on browser-based coding platforms. To keep pages responsive, computation can be offloaded to web workers or service workers, and the project ships examples for building Chrome extensions, from basic to advanced.

Model weight storage is configurable across four cache backends: the browser Cache API (the default), IndexedDB, the Origin Private File System, and an experimental cross-origin backend that requires a Chrome extension to be installed. If that extension is absent, WebLLM falls back to the default cache automatically. The documentation warns that the first model download can take significant time, so applications should treat loading as a long asynchronous operation; the engine provides an initialisation progress callback so developers can show download status to users.

Why it matters

Running inference on-device removes two of the biggest friction points in shipping web-based AI features: per-request inference costs for the developer, and data leaving the device for the user. A chatbot built on WebLLM keeps prompts and completions local, works offline once the model is cached, and costs the operator nothing beyond bandwidth for the initial download.

The milestone is also technical. Browsers were long considered too constrained for serious model serving, and a WebGPU-accelerated engine running 8B-class models competently changes that calculus. The limits remain real — multi-gigabyte first downloads, uneven GPU support across browsers and devices, and a smaller model catalogue than cloud APIs offer — but as a signal of where client-side AI is heading, WebLLM argues that the browser is now a legitimate deployment target for language models.