Azure: a blast-radius defect isolates an entire West US data center for nearly five hours

2026-07-23 · Publisher : Microsoft · Domain : Cloud & DevOps

On 23 July 2026, during a repair on an optical device, the blast-radius analysis system extends the maintenance to all egress devices of the West US data center. External traffic is cut from 14:44 to 19:41 UTC.

What changes

  • Outage on 23 July 2026, from 14:44 to 19:41 UTC, in the West US region.
  • Cause: the blast-radius calculation extended a repair to all egress optical devices.
  • Inbound and outbound traffic cut; traffic internal to the region intact.
  • Dated commitments: fixes done, monitoring in September, automated recovery in October 2026.

### What changes On 23 July 2026, from 14:44 to 19:41 UTC, **Azure** customers experienced connection failures, increased latency or difficulty accessing services in the **West US** region. Microsoft's post-incident review (PIR) explains the cause: during a repair on an optical device, a **defect in the blast-radius analysis system** "incorrectly expanded the scope of the repair event to include all optical devices egressing" the data center. The safety validation did not evaluate the aggregate effect of isolating all those devices at the same time. ### Key facts - Only the data center's inbound and outbound traffic was affected; traffic internal to the region remained intact. - Services affected: App Service, Application Gateway, AKS, Cosmos DB, ExpressRoute, Firewall, Sentinel, among others. - Manual rollback started at 17:45 UTC; devices recovered at 18:26; services fully recovered at 19:41. - Commitments: blast-radius calculation defect fixed, stronger checks to prevent simultaneous maintenance on redundant paths (done); better traffic monitoring (September 2026) and more robust automated recovery (October 2026). ### Why it matters The incident illustrates a risk SRE engineers know well: the tool meant to make a change safe itself widened its impact. The provider's maintenance path remains an invisible dependency for customers. For IT leaders: test scenarios of losing a region's external traffic, and require a detailed PIR with dated commitments from every provider, as here. ### Takeaway A poorly calibrated maintenance safety check isolated an entire data center: change safety is a platform issue.

Links

Tags : Incident, Azure, Hyperscaler, Cloud