www.ably.io
Back

Incident Details

All times are shown in UTC

9th October 2020 02:38:25 PM

Potential pre-emptive action for AWS US-East capacity problems

AWS have been reporting issues with new instances coming online in the US East region.

This has had no impact on our service in this region however we are actively monitoring it. If the situation changes such that we are not confident there is enough capacity to service the traffic, we will likely route traffic away from US East until AWS EC2 instances are stable in that region.

9th Oct 08:14 PM

Amazon reported that they have now resolved the issues in US East 1, and we have resumed normal operations again in all regions.

The update from AWS on the root cause of this problem is as follows:

Starting at 9:37 PM PDT on October 8th, we experienced increased network connectivity issues for a subset of instances within a single Availability Zone (use1-az2) in the US-EAST-1 Region. This was caused by a single cell within the subsystem responsible for the updating VPC network configuration and mappings experiencing elevated failures. These elevated failures caused network configuration and mappings to be delayed or to fail for new instance launches and attachments of ENIs within the affected cell. The issue has also caused connectivity issues between an affected instance in the affected Availability Zone(use1-az2) and newly launched instances within other Availability Zones in the US-EAST-1 Region, since updated VPC network configuration and mappings were not able to be updated within the affected Availability Zone(use1-az2). The root cause of the issue was addressed and at 10:20 AM PDT on October 9th, we began to see recovery for the affected instances. By 11:10 AM PDT, all affected instances had fully recovered. The issue has been resolved and the service is operating normally

Resolved

in about 7 hours