· via Hacker News – Front Page (native)
Guide walks through running free local LLMs at home with Unsloth Desktop and Qwen
A GPUPartner guide walks through setting up a private, free local AI on a Windows PC with Unsloth Desktop and quantized Qwen or Inkling models, with no subscription or terminal commands required.
A guide published by GPUPartner on November 22, which reached Hacker News's front page, walks beginners through building a private, household AI assistant on a Windows PC using the free Unsloth Desktop application and open-weight models such as Qwen and Inkling. The author reports that once the hardware is sorted, configuration takes roughly 10 to 15 minutes, not counting model downloads.
The stated motivation is frustration with hosted AI services. The author wanted to offload token-heavy research, data collection and coding chores without watching usage caps all day, and argues that OpenAI and Anthropic have shipped quality regressions in their newest flagship models while tightening control over what those models will do. That is the author's characterization rather than a benchmarked finding, but it frames the practical case being made: cost, privacy and rate limits pushed them toward self-hosting.
A three-step setup
The walkthrough keeps things deliberately simple, and it targets Windows because this was the author's first attempt. Step one is installing Unsloth Desktop, a free application that serves a model through a built-in chat interface. Step two is downloading a quantized model from a dropdown menu. Step three is setting a context size, or leaving it on automatic, and enabling LAN access so everyone in the home can reach the assistant. No terminal commands, Python environments or subscriptions are required.
The author considered LM Studio as an alternative and picked Unsloth Desktop partly because its site did not push a subscription, the very thing being escaped, and partly on positive community sentiment.
Matching a model to your GPU
How much video memory a graphics card has determines what it can run. As a rough guide, the author says 8 to 12 GB handles smaller 9B to 12B models, while 32 GB comfortably runs 27B to 35B models with room left for long conversations. The full-size Qwen model sits around 56 GB, which is where quantization comes in: compressing weights to lower precision shrinks the model enough to fit on a consumer card and run faster, at some cost in accuracy. Whether that trade is worth it depends on the workload, since some tasks give similar outputs while others may become nearly unusable.
On a test machine with an RTX 5090 and 32 GB of VRAM, the guide uses Qwen3.8-27B with Unsloth's 4-bit UD-Q4_K_XL build, deliberately choosing a 128K context window instead of cramming in the largest model that fits. The author wryly notes that even an expensive card like the 5090 remains a comparative deal next to enterprise GPUs. For smaller cards, the guide suggests Qwen3.5-9B or Gemma 4 12B at 8 to 12 GB, Qwen3-14B or gpt-oss-20b at 16 GB, and Qwen3-Coder-30B, Gemma 4 26B/31B, Qwen3.8-27B or Inkling 2 Mini at 24 GB, with 32 GB-plus cards supporting 64K to 128K+ contexts. GPUPartner also publishes a VRAM calculator for estimates.
Macs work too, thanks to unified memory, though the guide says performance trails discrete GPUs; users should grab the MLX builds made for Apple Silicon. Unified-memory PCs such as the MSI PRO MAX EDGE AI+ 11M and NVIDIA's DGX Spark sit in the same category.
Inkling 2 Mini joins the list
The guide highlights Inkling 2 Mini, which Thinking Machines Lab released on November 17, 2026: 26.4 billion parameters, an Apache 2.0 license, vision support and a 1 million-token context. Its recommended 4-bit file is 17.6 GB, within 0.1 GB of the Qwen3.8-27B file used in the guide, so it occupies a 32 GB card the same way and leaves roughly 14 GB for context. Per Unsloth's figures, the 4-bit model fits a 24 GB card, while the lossless 8-bit build needs about 24 GB and a 32 GB card; context adds on top.
Inkling 2 Small is a different story. At 594 billion parameters, its smallest published file is 171.7 GB, which is unified-memory territory such as a 256 GB Mac Studio or a DGX Station rather than a graphics card.
Why it matters
The guide is a snapshot of how far open-weight models have come: a single consumer GPU can now host a private assistant capable of routine work, with no metering, no subscription and no vendor deciding what the model may say or do. In the author's framing, cloud LLM providers start to look like plain infrastructure for everyday tasks. The limits remain real, since quantization degrades some workloads and frontier-scale weights still demand datacenter-class hardware, but for the research, summarization and coding chores that fill most days, leverage is shifting toward whoever holds the GPU.
- #local-llm
- #unsloth
- #qwen
- #quantization
- #open-weights