Cloudflare loses 55% of customers' logs after a misconfiguration

2024-11-14 · Publisher : Cloudflare · Domain : Cloud & DevOps

On 14 November 2024, a configuration bug at Cloudflare overloads its log pipeline: about 55% of the logs normally sent to customers are lost over roughly 3.5 hours.

### Background On 14 November 2024, Cloudflare suffered an incident on its **Logs** and **Logpush** services, which deliver traffic logs to customers' tools (SIEM, observability). The incident lasted about **3.5 hours** and, according to the post-mortem published on 26 November, "about 55% of the logs we normally send to customers were not sent and were lost". ### Key facts - Cause 1: a bug in the configuration system produced a blank configuration for the Logfwdr component, indicating that no customer had logs to push. - Cause 2: a "fail open" safeguard in Logfwdr reacted by forwarding logs for all customers, not only the configured ones. - Cause 3: the Buftee buffering component lacked a sufficient safeguard and created about 40 times more buffers than normal, until overload. - Remediation: alerts on specific misconfigurations, a fix for the bug with tests, and regular "overload tests" simulating cascading failures. ### Why it matters Logs are the raw material of security detection and compliance evidence. A silent loss of more than half of them on the provider side leaves gaps the customer cannot fill after the fact. For SRE teams, it is a reminder that the observability chain is itself a system to monitor: alerts on volumes received, not only on volumes sent. ### Takeaway A failing observability pipeline raises no alert: the data loss only shows if one compares what arrives with what should arrive.

Tags : Incident, Observabilité, Edge, Sécurité