From-scratch C++ and CUDA LLM inference engine ships with video course and handwritten notes

A teaching engine, not a product

A project that reached Hacker News's front page, samir-vllm is a small LLM inference engine written from first principles in C++17 and CUDA. Its stated purpose is education: the repository traces every step between a prompt arriving and a token being emitted, from parsing weight files to running each transformer component on the GPU. According to the project's README, the code is paired with a YouTube build series of more than six hours, covering an intro to inference, weight loading, embedding gather, RMSNorm, RoPE, Q/K/V projections via cuBLAS, self-attention, the feed-forward network, decode and the KV cache, static and continuous batching, and a paged KV cache, before closing with paged attention. A companion PDF of more than 80 pages of handwritten notes and diagrams comes with markdown sources in the repository, and the author states the notes were written without AI assistance.

What the engine implements

The engine targets a single model, Llama 3.2 1B Instruct, with a fixed tensor layout across its 16 layers. It loads model.safetensors directly, parsing metadata, validating tensor offsets and copying raw BF16 weights to GPU memory, and also offers a memory-mapped loader to skip an extra CPU-side copy. Tokenization is handled locally with a BPE tokenizer that supports Unicode through the ICU library, special tokens and the Instruct chat prompt format, and it ships with a dedicated test executable.

On the compute side, the dense causal prefill path runs all 16 layers: Q/K/V projection, RoPE, grouped-query attention, output projection, the attention residual, a post-attention RMSNorm, a SwiGLU MLP and the MLP residual. Custom CUDA kernels cover embedding gather and RMSNorm, the latter reducing in FP32 over the 2,048-wide hidden state while applying BF16 learned weights. A final RMSNorm and a tied embedding/LM-head projection produce FP32 logits, and a GPU argmax picks each next token. Generation is greedy, with end-of-turn and end-of-text stopping inside a 2,048-token context.

KV paging and scheduling

The memory design mirrors what production serving stacks do. A shared BF16 KV page pool backs a block table per sequence and per layer, with each physical page holding 16 tokens of K and V. Prefill fills a request's prompt pages, and decode appends one new token at a time at its absolute RoPE position. The pool defaults to 128 MiB and can be resized with a command-line flag.

Two scheduling modes are implemented: static batching waits for the current group to finish before admitting more, while a continuous scheduler admits queued prompts whenever page budgets allow and releases their pages on completion. Admission reserves room for a request's worst-case processed prefix, prompt tokens plus the allowed new tokens minus one, rounded up to 16-token pages in each of the 16 layers. Requests that cannot fit wait in FIFO order, and a request whose budget exceeds the entire pool is rejected with an error. The README notes this conservative policy avoids preemption and recomputation, at the cost of some concurrency when large output budgets are declared. Decode attention streams each query head's cached pages with an FP32 online softmax, accumulating results without copying the cache into a contiguous buffer or storing full attention score matrices.

Caveats and requirements

The README is candid about limitations. Model dimensions and the 2,048-token context are fixed, prefill still handles one prompt at a time between decode iterations, and because BF16 arithmetic can round differently at different batch sizes, greedy outputs may diverge between individual and batched runs; bitwise equality is not guaranteed. Building requires an NVIDIA GPU with the CUDA Toolkit, CMake 3.24 or newer, a C++17 compiler, ICU development libraries, the nlohmann/json header, and user-supplied Llama 3.2 1B Instruct weights and tokenizer. The engine source is MIT-licensed; the notes carry separate terms.

Why it matters

Serving engines make inference look like a single API call, but underneath sit weight loading, custom kernels, KV cache paging, online-softmax attention and continuous batching, the same techniques used by production stacks such as vLLM. This project rebuilds all of those pieces in miniature on a 1B-parameter model. For engineers who tune latency, cost or GPU utilisation, working through a complete, runnable implementation is one of the few ways to see how the parts interact, and the pairing of code, video and handwritten notes makes it usable as a self-contained curriculum rather than just a reference codebase. It is explicitly a learning engine rather than a production one, but the concepts transfer directly to the systems running real workloads.