CrowdStrike: a faulty update takes 8.5 million Windows PCs down
2024-07-19 · Publisher : CrowdStrike · Domain : Cloud & DevOps
On 19 July 2024, a CrowdStrike Falcon content update causes blue screens on Windows machines. Microsoft estimates 8.5 million devices were hit. The faulty file was live for 78 minutes; getting machines back took far longer.
What changes
- Update pushed at 04:09 UTC, reverted at 05:27 UTC: 78 minutes of exposure.
- Microsoft estimates 8.5 million Windows devices hit, less than 1% of the Windows estate.
- Cause: 21 fields in a content template, only 20 validated; out-of-bounds memory read.
- CrowdStrike promises staged rollouts and more customer control.
### Background On Friday 19 July 2024 at 04:09 UTC, **CrowdStrike** pushed a content update to its Falcon sensor ("Rapid Response Content", channel file 291). A defect missed by its validation tool caused, on Windows sensors running version 7.11 and above, an out-of-bounds memory read, hence a crash and a blue screen. The update was reverted at 05:27 UTC, about **78 minutes** later. Mac and Linux machines were not affected. ### Key facts - Microsoft estimates **8.5 million** Windows devices were affected, less than 1% of all Windows machines. Each machine had to be repaired individually, often by hand. - Microsoft deployed hundreds of engineers and worked with AWS and Google Cloud on remediation; repair procedures and scripts were published. - Root cause analysis published on 6 August 2024: a content template had 21 fields while only 20 were validated, hence the out-of-bounds read. - Six commitments, including compile-time validation, a runtime bounds check and **staged deployment** of future content, with customer control over update timing. ### Why it matters The outage did not come from a public cloud but from a security agent installed in the kernel of millions of Windows workstations and servers (Microsoft worked with AWS and Google Cloud on remediation). It shows that a trusted vendor is also a single point of failure. For IT leaders: demand progressive rollouts for agents, plan recovery procedures that work without network access, and include security vendors in dependency analyses. ### Takeaway A single update, live for 78 minutes, was enough to immobilize millions of machines, and made staged rollouts a purchasing criterion.
Links
Tags : Incident, Sécurité, Cloud, États-Unis