OpenAI: a new telemetry service overloads Kubernetes and takes ChatGPT down for over four hours

2024-12-11 · Publisher : OpenAI · Domain : Cloud & DevOps

On 11 December 2024, the rollout of a telemetry service overloads OpenAI's Kubernetes API servers and breaks DNS-based service discovery. ChatGPT, the API and Sora are degraded for 4 hours 22 minutes.

### What changes On 11 December 2024, between 3:16 pm and 7:38 pm (Pacific time), OpenAI's ChatGPT, API and Sora were unavailable or degraded. The company's post-mortem gives the cause: a freshly deployed **telemetry service** generated unexpected load on the **Kubernetes API servers** of large clusters. Every node in a cluster ran resource-intensive API operations whose cost scales with the size of the cluster. The control plane was overwhelmed, and DNS-based service discovery stopped working. ### Key facts - Duration of significant degradation: 4 hours 22 minutes. ChatGPT fully recovered at 7:01 pm, the API at 7:38 pm, Sora at 7:01 pm. - Testing in smaller staging clusters did not reveal this scale-dependent defect. - DNS caching delayed visibility of the problem, which let the rollout continue across the whole fleet. - Once the control plane was saturated, access to it for repairs became impossible. - Measures: phased rollouts with monitoring, fault-injection testing, emergency access to API servers, decoupling the data plane from the control plane. ### Why it matters The case illustrates a classic trap: the observability tool itself can cause the outage. For platform teams, it means testing agent rollouts at real scale, limiting their blast radius, and keeping a fallback access to the Kubernetes control plane that does not depend on the cluster's DNS. ### Takeaway A monitoring agent rolled out too fast took down the control plane it was meant to watch.

Tags : Incident, Kubernetes, Observabilité, DevOps