AWS us-east-1 outage: a race condition in DynamoDB's DNS cripples the region for 14 hours

2025-10-20 · Publisher : AWS · Domain : Cloud & DevOps

In the night of 19-20 October 2025, a DNS automation defect empties the address of the DynamoDB endpoint in us-east-1. EC2, Lambda, load balancers and other services are disrupted for about 14.5 hours.

### Background From 19 October 2025 at 11:48 pm to 20 October at 2:20 pm (Pacific time), about **14.5 hours**, **AWS** suffered a major outage in its **us-east-1** region (Northern Virginia). The official account gives the cause: a **latent race condition** in DynamoDB's automated DNS management system. Two independent components ("DNS Enactors"), running in different availability zones, overlapped: an older DNS plan overwrote a newer one, then was deleted, which **removed all IP addresses** of DynamoDB's regional endpoint. Manual intervention was required. ### Key facts - Services affected: DynamoDB first, then EC2 instance launches, Network Load Balancers, Lambda, ECS, EKS, Fargate, Amazon Connect, Redshift, STS and management console authentication. - Three distinct impact periods, with cascading failures. - Announced measures: disable DynamoDB DNS automation worldwide while fixes are made, repair the race condition, add velocity controls to NLB health-check failovers, expand EC2 scale testing and improve throttling of propagation flows. - AWS apologizes to customers in the account. ### Why it matters A managed database is a hidden dependency of many other services: when its name resolution disappears, everything calling it falls, administration tools included. For IT leaders, the lesson is to design for a degraded us-east-1 (active multi-region, or at least a rehearsed recovery plan), to reduce control-plane dependencies in a crisis and to test the loss of DNS. ### Takeaway An automation defect cut access to DynamoDB; the domino effect was far wider than the database itself.

Tags : Incident, AWS, Hyperscaler, Cloud