Amazon reported that they have now resolved the issues in US East 1, and we have resumed normal operations again in all regions.

The update from AWS on the root cause of this problem is as follows:

Starting at 9:37 PM PDT on October 8th, we experienced increased network connectivity issues for a subset of instances within a single Availability Zone (use1-az2) in the US-EAST-1 Region. This was caused by a single cell within the subsystem responsible for the updating VPC network configuration and mappings experiencing elevated failures. These elevated failures caused network configuration and mappings to be delayed or to fail for new instance launches and attachments of ENIs within the affected cell. The issue has also caused connectivity issues between an affected instance in the affected Availability Zone(use1-az2) and newly launched instances within other Availability Zones in the US-EAST-1 Region, since updated VPC network configuration and mappings were not able to be updated within the affected Availability Zone(use1-az2). The root cause of the issue was addressed and at 10:20 AM PDT on October 9th, we began to see recovery for the affected instances. By 11:10 AM PDT, all affected instances had fully recovered. The issue has been resolved and the service is operating normally

We have temporarily redirected traffic away from us-east-1 to other datacenters

The issue is due to AWS issues with resolving dns from within us-east-1: https://status.aws.amazon.com/ . We are leaving traffic redirected away from us-east-1 until that is resolved. (Any connections that were already connected to us-east-1 will remain for now)

Following AWS reporting the dns resolution latency issues as fixed on their end, which we have confirmed, we are now re-enabling traffic to us-east-1.

Total customer-visible effect should have been very little, other than occasional latency spikes for requests to us-east-1 in the time between the AWS issue beginning and when we redirected traffic away from that region at 08:15 UTC, and fractionally higher latencies for customers near us-east-1 who were redirected to us-west-1 for the duration.

Within 20 minutes of active management the load on our global database has returned to normal levels.

Our initial investigation indicates that customers using our Push registration and delivery APIs were most affected during this period.

We are investigating the root cause of this issue now and will continue to post updates as we know more.

We apologise for any inconvenience this may have caused.

DNS migration is now complete with 3rd party provider Freshdesk.

A fix for this problem is being deployed now and we are monitoring the situation.

The service is back to normal.

Our engineering and ops teams have completed the post mortem of this incident and summarised all actions we have taken to ensure we can avoid any future disruption to our service.

See https://gist.github.com/paddybyers/c27d302524caa8e46f41e9ba19fdcf2e

Heroku seems to be having ongoing issues; there's no explanation in https://status.heroku.com/incidents/1973. We will continue to monitor the situation.

A downstream provider of our database and web services performed maintenance on our database earlier today, which required a change in the endpoint used by all web services. Unfortunately the update was only made to one of the two web services required, which caused the async operations to fail during this period.

The issue is now fully resolved, and we'll be investigating why this update was only applied to one of the two web services.

As yesterday, the incident resolved itself after 9 minutes. We continue to investigate as a top priority.

We have now identified the root cause of the recent latency issues we've experienced in the global persistence layer, and have rolled out updates in the global persistence layer that have ensured that the latencies are consistently low.

The primary cause of the problem was an inadequate rate limiter in one area of our system, which allowed our persistence layer to be overloaded and thus impact the global service latencies for operations that rely on the persistence layer (primarily history, push registrations, and persisted tokens).

A full post mortem will follow soon.

Our engineering and ops team have completed the post mortem of this incident and summarised all actions we have taken to ensure we can avoid any future disruption to our service.

See https://gist.github.com/pauln-ably/03098db1095f4ef61aac801ae987dac2

Following more than 36 hours of intermittent short and significant increases in latencies in our persistence layer, the engineering team have been investigating the root cause to understand why only a small percentage of shards are affected during this time.

We have made significant progress in identifying potential root causes, however in the meantime we have also been addressing the issues by adding capacity and upgrading the entire cluster.

The persistence cluster is now operating with approximately 3x more capacity than it had 24 hours ago and is now upgraded to the latest stable versions.

We'll continue to investigate what triggered these increases in latencies, however we are optimistic that the increased capacity will now offer stability and predictable performance moving forward.

A full post mortem will be published soon.

Please see https://status.ably.io/incidents/695 for the post mortem of this disruption.

Latencies are back to normal as of 11:21 UTC

Please see https://status.ably.io/incidents/695 for the post mortem of this disruption.