Run local LLMs from Spring Boot via Docker Model Runner's OpenAI-compatible API

Java developers can run large language models on their own machines and reach them from a Spring Boot application through Spring AI, without routing prompts to a hosted provider such as OpenAI or AWS Bedrock. A tutorial published on dev.to walks through the setup and argues that this removes cloud dependencies during development while leaving the application code largely untouched.

How the stack fits together

The tutorial describes a layered setup: the Spring Boot application talks to Spring AI, which issues calls against an OpenAI-compatible API, which Docker Model Runner serves on the local machine, backed by a locally running model. The key detail is compatibility rather than provenance — an OpenAI-compatible endpoint simply follows the same request and response format as OpenAI's API, so pointing an OpenAI client at localhost is enough to reach a local model.

The case for local inference

The author lists several motivations. Development often involves hundreds or thousands of prompts, and a local model eliminates API charges during experimentation. Prompts and application data stay on the developer's machine instead of going to an external provider, which matters when working with sensitive or proprietary material. Once the model is available locally, inference works without an internet connection, and developers can iterate on prompts, tool calling, RAG pipelines and application logic without repeatedly configuring cloud credentials.

Perhaps the biggest advantage, according to the tutorial, is that the Spring AI programming model stays the same. Business logic is not tightly coupled to a specific provider, so the abstraction carries over regardless of where the model runs.

Wiring it up in Spring Boot

The walkthrough proceeds in five steps:

1. Install and start Docker Desktop. Model Runner availability depends on the Docker Desktop version and configuration, and the author notes that the exact commands and model catalogue may change as Docker's ecosystem evolves, so current Docker documentation is the reference point.
2. Enable Model Runner in Docker Desktop, which exposes an API endpoint the application can target.
3. Create a Spring Boot project and add the spring-ai-starter-model-openai Maven dependency. This starter is chosen for its wire format, not because OpenAI's cloud service is being called.
4. Configure Spring AI's OpenAI base URL to point at the local Model Runner endpoint, set the chat options model property to the local model's name, and supply a placeholder API key — the local runtime may not require one, but the Spring AI client expects the property to be present.
5. Build a ChatClient.

For the application code, the tutorial injects the autoconfigured ChatClient.Builder into a REST controller, builds the client in the constructor, and exposes a GET endpoint that takes a message parameter and returns the response from a simple prompt call. The result is a URL along the lines of a chat endpoint with a message query parameter, with the request flowing from HTTP through Spring Boot, ChatClient and Spring AI to Docker Model Runner and the local model.

The provider disappears behind the ChatClient

Because the controller only sees the ChatClient abstraction, it does not need to know whether the backend is OpenAI, Azure OpenAI, AWS Bedrock, Ollama, Docker Model Runner or another compatible provider. The application states what it wants from the model, and Spring AI handles the underlying communication. In practice, swapping providers becomes a configuration change rather than a rewrite.

Local versus cloud trade-offs

The tutorial avoids overselling local execution. Local models keep data on the machine and avoid per-request charges, but they consume local compute, may run more slowly and are bounded by the available hardware. Cloud offerings handle the infrastructure, tend to be faster and give access to much larger models, usually under usage-based pricing. A developer laptop can comfortably host a smaller model, while a large frontier model demands considerably more memory and compute. The question the author poses is not whether local beats cloud, but which deployment strategy fits a given workload — local for development, cloud for production.

Why it matters

The combination lowers the barrier for Java teams experimenting with generative AI: no cloud account, no per-prompt spend and no data leaving the machine during development. It also illustrates a broader pattern in the ecosystem — OpenAI-compatible APIs have become a de facto interface, so any runtime that implements the format can slot into tooling originally built for OpenAI. For teams already standardised on Docker, Model Runner brings model execution alongside the rest of the local development environment, and because Spring AI insulates the application from the provider, a later move to a hosted model is mostly a matter of editing configuration rather than reworking code.