llm-d is donated to the CNCF as a Sandbox project at KubeCon Amsterdam

2026-03-24 · Publisher : Red Hat · Domain : Cloud & DevOps

On 24 March 2026, at KubeCon Europe, IBM Research, Red Hat and Google Cloud donate llm-d to the CNCF as a Sandbox project, ten months after its launch: a Kubernetes distributed-inference framework backed by NVIDIA, CoreWeave, AMD, Cisco, Hugging Face and others.

### Background On 24 March 2026, at KubeCon + CloudNativeCon Europe, **Red Hat, IBM Research and Google Cloud** donate **llm-d** to the CNCF at the **Sandbox** level. Launched in May 2025, the project is a distributed language-model inference framework on Kubernetes. Founding contributors NVIDIA and CoreWeave, along with AMD, Cisco, Hugging Face, Intel, Lambda and Mistral AI, accompany it; UC Berkeley and the University of Chicago support it. ### Key facts - A data-plane orchestration layer between control planes and inference engines, specialized for LLMs. - An endpoint picker (EPP) routes requests according to KV cache hit rates and hardware characteristics. - Native Kubernetes integration through Gateway API and LeaderWorkerSet primitives. - Metrics export to Prometheus and Grafana, including time to first token. - Problem addressed: the gap between experimentation and production inference, where standard routing ignores the variable cost of requests by token length and processing phase. ### Why it matters Joining the CNCF positions the foundation as the place for standardizing inference, after AI Conformance and KServe's incubation. For an IT services firm, the signal is twofold: a Kubernetes inference pattern backed by several chip and cloud vendors becomes more likely as a reference, and neutral governance reduces lock-in risk. Sandbox remains the first level: the project does not yet have the production track record of mature projects. ### Takeaway llm-d enters the CNCF: distributed LLM inference looks for its open standard on Kubernetes.

Tags : Kubernetes, IA, Open Source, NVIDIA, Google