Modelplane publishes open field guide to running LLM inference in production

Modelplane has published an open-access field guide to inference engineering, written for platform engineers who are new to running large language models on their own GPUs. The document appeared on Hacker News's front page on 10 November 2026, and its numbers, prices and tool versions are current as of October 2026. Modelplane is built by Upbound, the company behind Crossplane.

Who the guide is for

The guide defines the job as picking a model, getting it onto suitable hardware, keeping it serving, making it faster and cheaper, and being on call when it fails — essentially site reliability work for a new class of workload. It openly builds on the wider inference community, citing work from Viktor Farcic, Philip Kiely, The Pragmatic Engineer, ByteByteGo, Data Gravity, Spheron, Vishakha Sadhwani and Upbound's own Project Champagne series.

The case for self-hosting

According to the guide, platform teams inside AI-heavy organizations are turning into inference teams. Coding assistants cannot send source code to third parties, frontier model bills climb every month, and internal support tools need data that hosted APIs cannot reach. Meanwhile, Hugging Face now hosts more than two million open models, and the strongest of them rival closed ones — DeepSeek's R1, released in January 2025, matched OpenAI's o1 on math and coding benchmarks roughly a month after o1 shipped.

Running your own model means controlling latency, uptime and where data goes, and at high volume can cut costs by around 80%, the guide claims. It also argues that the price of a fixed level of AI capability drops roughly tenfold a year, and that most of that comes from how models are served rather than from better GPUs. Upbound's own experience bears this out: during Project Champagne, running models on Modelplane, the vLLM engine and scheduling worked fine, while the real effort went into TLS, identity, timeouts, storage, IAM and observability.

The guide is candid about timing, though. A hosted API is still the right choice early in a product's life; self-hosting pays off once API spend becomes a real budget line, when latency or reliability requirements outgrow vendors, or when data cannot leave the network. It also notes that Chris Zeoli counted only about 58 open inference roles on public careers boards in August 2026, with no settled job title.

Skills that carry over

Much of the guide maps existing platform knowledge onto inference. Node scheduling becomes GPU placement with device selectors and exclusive devices. Capacity planning becomes GPU memory math covering weights, KV cache and runtime overhead. Load balancing becomes cache-aware routing over long-lived streaming connections, and autoscaling keys off queue depth and KV cache pressure, with cold starts measured in minutes. SLOs shift to time to first token, time per output token and goodput, while FinOps tracks cost per million tokens and per user session.

The mental model

An inference engine such as vLLM or SGLang is the server that exposes model weights as an API. The guide's closest analogue is a database server: it loads a large dataset into memory at startup, keeps per-session state, schedules work internally and answers over HTTP. The big difference is that its working set must sit in small, expensive GPU memory rather than system RAM, which is what most tuning is about. Ollama on a laptop is the same idea at small scale, and every hosted model API runs one behind the scenes. Above the engine sit kernels doing the arithmetic, while a control plane deploys and manages engines and a gateway routes traffic across models and replicas.

Memory math and common mistakes

Weights are the fixed footprint: most models ship at two bytes per parameter, so an 8B model needs about 16 GB of GPU memory, leaving roughly two-thirds of a 48 GB L40S for everything else. The KV cache is the working set, and the guide gives a formula for estimating it from a model's config. For Llama 3 70B at FP16, that works out to about 320 KB per token, meaning a single 128K-token conversation needs around 40 GB of cache — often the real limit on how many users one GPU can serve.

Three sizing traps are flagged. Mixture-of-experts models keep every expert resident, so a model with 320B total and 18B active parameters still needs about 320 GB at FP8. You should read the checkpoint size rather than the dtype table: Upbound estimated one MoE model at 315 GB from its dtypes, but the actual checkpoint was 177.7 GB because experts were stored in 4-bit form. And runtime overhead from CUDA graphs, activations and fragmentation means a model that fits at startup can still exhaust memory days later. Each request also runs in two phases — a compute-bound prefill that reads the whole input and sets time to first token, followed by decode, which writes the answer one token at a time.

Why it matters

As open models close the gap with hosted frontier APIs, serving them is becoming a standard platform-engineering responsibility rather than a research specialty. This guide consolidates scattered community knowledge into one practical reference with concrete memory arithmetic, which is genuinely rare. Just as importantly, it shows that the hard parts are familiar ones — networking, identity, observability and cost control — meaning the many engineers already running stateful services in production are far closer to inference roles than they might assume.