Self-hosting an LLM sounds like a model choice. In practice, it quickly becomes a platform choice.

The useful part of CNCF's new walkthrough on running vLLM in Kubernetes is not that it proves every team should run its own models. It does not. Managed model APIs are still the right answer for many workloads. The interesting part is the operational shape it documents: if a team wants local inference for cost predictability, latency control, data locality, or high-volume internal workloads, the model server has to behave like production infrastructure.

That means more than starting a container. The lab setup uses Kubernetes as the orchestration layer, vLLM as the inference engine, LINSTOR through the Kubernetes CSI path for persistent storage, and an OpenAI-compatible HTTP surface so existing clients can point at the local endpoint without rewriting the application.

The shift is simple: private inference is not a notebook anymore. It is a service with storage, identity, networking, restart behavior, and capacity planning.

The Model Is Only One Component

The CNCF post uses meta-llama/Llama-3.2-1B-Instruct, a small instruction-tuned model that can run in a CPU-only lab. That makes the example approachable, but it also exposes the infrastructure problem clearly. Before the model can answer anything, the cluster needs a way to fetch weights, store them across restarts, pass the Hugging Face token safely, run the vLLM process, and expose the service inside the cluster.

The article's core Kubernetes objects are intentionally ordinary: a PersistentVolumeClaim for model storage, a Secret for the token, a Deployment for the inference server, and a Service to give clients a stable network target. The model cache is mounted under /root/.cache/huggingface, which means the first startup can download the model weights, while later restarts avoid paying that cost again.

That is the part platform teams should notice. Model weights are not just files. They are large, slow-to-refresh, access-controlled runtime dependencies. If every restart turns into another download, your model server has an availability problem. If the weight cache lives only inside the pod, your cluster has made the model disposable in the wrong place.

model weights\n    -> persistent volume\n    -> vLLM deployment\n    -> Kubernetes service\n    -> OpenAI-compatible endpoint\n    -> application clients
A private model endpoint still needs the same boring platform pieces as any other production service.

Why vLLM Fits The Boundary

vLLM is built for inference and serving rather than training. Its own documentation describes it as a fast, easy-to-use library for LLM inference, with PagedAttention, continuous batching, prefix caching, structured outputs, streaming, tool-calling support, and an OpenAI-compatible API server. That API compatibility is not a cosmetic feature. It is what lets an application treat a local model endpoint as part of a hybrid architecture rather than as a one-off experiment.

If code already talks to an OpenAI-style chat completions endpoint, the platform can route some traffic to a managed provider and some to an internal vLLM service. The decision can be based on cost, latency, sensitivity, context length, model capability, or reliability. CNCF's post points toward that next step with inference routing, including the open source llm-d project for Kubernetes.

The important design move is keeping the application boundary stable. The fewer assumptions application teams make about where inference runs, the easier it becomes to shift workloads. A local model can handle high-volume summarization, internal classification, test fixtures, or privacy-sensitive helper tasks. A managed frontier model can still handle requests where capability matters more than locality.

Storage Is Part Of The Inference Stack

The walkthrough uses LINSTOR because it provides replicated block storage through the Container Storage Interface. The specific storage product is less important than the pattern: model serving needs persistent state even when the application is logically stateless. The pod can be replaced. The service can be stable. The downloaded weights should not vanish every time the scheduler moves the workload.

This is where AI infrastructure starts looking like ordinary infrastructure again. A reliable model endpoint needs a startup path, a cached artifact path, a secret path, a scaling path, and a failure path. In the CNCF lab, scaling the deployment down to zero preserves the persistent volume and cached model weights, so scaling it back up does not require another model download. That is not glamorous. It is exactly the kind of behavior operators need before they trust a service.

GPU nodes make the performance story more serious, but they do not change the shape of the problem. They add scheduling pressure, cost pressure, device plugin behavior, memory limits, and utilization questions. The platform contract still has to answer the same basic questions: where are the weights, who can access them, how does traffic find the server, what happens on restart, and how do clients fail over?

The Takeaway

The best reading of this CNCF post is not that every enterprise should build a private model cloud. It is that self-hosted inference is becoming legible to platform teams. It has resources, manifests, volumes, secrets, endpoints, and operational tradeoffs that can be reviewed instead of hand-waved.

That matters because hybrid AI is likely to be messy. Some tasks will stay with managed APIs. Some will move to local open models. Some will need routers that compare latency, price, data locality, and quality before choosing where to send a request. The teams that do well will be the ones that treat model serving like a normal service early, with the same discipline they already apply to databases, queues, and internal APIs.

The model is the visible part. The platform is what decides whether it can keep answering tomorrow.

Sources