Red Hat launches llm-d, a Kubernetes stack for distributed LLM inference

2025-05-20 · Publisher : Red Hat · Domain : Cloud & DevOps

On 20 May 2025, at Red Hat Summit, Red Hat launches llm-d with CoreWeave, Google Cloud, IBM Research and NVIDIA: an open source distributed inference project on Kubernetes, built on vLLM, with KV cache and AI-aware routing.

### Background On 20 May 2025, at Red Hat Summit in Boston, Red Hat launches the **llm-d** community, an open source project for large-scale distributed language-model inference on Kubernetes. Founding contributors are CoreWeave, Google Cloud, IBM Research and NVIDIA; AMD, Cisco, Hugging Face, Intel, Lambda and Mistral AI are partners, with support from UC Berkeley (Sky Computing Lab) and the University of Chicago (LMCache Lab). ### Key facts - **vLLM** as the open source inference server, compatible with many accelerators, including Google Cloud TPUs. - Prefill and decode phases separated onto different servers, to spread the load better. - KV cache offloading from the GPU to CPU memory or network storage, via LMCache. - "AI-aware" network routing: requests go to servers that already hold the relevant inference cache. - High-performance communication interfaces, with support for NVIDIA's NIXL library; scheduling on Kubernetes clusters. ### Why it matters Inference workloads do not behave like ordinary web services: the cost of a request depends on token length and processing phase, which defeats standard load balancing. llm-d offers a common, cloud-provider-neutral layer to address it. For an IT services firm, it is an architecture reference for internal AI platforms, to be weighed against hyperscalers' managed services. For Brian Stevens (Red Hat), llm-d paves the way for distributed, high-performing inference across hybrid cloud. ### Takeaway llm-d proposes a common Kubernetes pattern for serving LLMs at scale, backed by five major players from day one.

Tags : Kubernetes, IA, Open Source, NVIDIA, Google