Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

Due to a load spike, message transit latencies for messages through the Asia Singapore datacenter may have been slower than normal for a period of around 10 minutes. The issue resolved itself automatically through autoscaling.

Cloudflare is back up, so fallback hosts are now responding as normal.

All Cloudflare-mediated endpoints have been moved away from Cloudflare.

Fallback endpoints are now restored, circumventing Cloudflare

We are investigating the root cause of the issue. If this issue is not resolved soon we will temporarily redirect traffic away from the us-west-1 datacenter until the underlying issues are resolved

All intermittent timeouts in the us-west-1 region have stopped since 09:21 UTC.

We believe the underlying issue was a networking issue in the underlying AWS datacenter, but have not been able to confirm that. However, for the last hour, the datacenter appears to be healthy.

Clients closest to us-west-1 experiencing timeouts would have been automatically reconnected to an alternative datacenter with our automatic fallback capability. See https://support.ably.io/a/solutions/articles/3000044636 for more details.

The us-east-1 (North Virginia) datacenter is healthy and active again. Traffic is now being routed to this datacenter as normal.

The ap-southeast-1 region is now fully operational once again.
We're continuing to investigate the underlying issue.

The faulty node has now been fully removed from the cluster, and all data has been successfully replicated to a new healthy node.

We have identified the underlying issue (dependency on a third party system that, by design, should not have impacted our ability to add capacity, but did due to an internal bug). We have applied hot fixes to all environments and rolled this out globally. Error rates are dropping rapidly and latencies reducing, however there are still some residual issues we are manually resolving.

We are still experiencing issues in us-east-1 which is causing higher than normal error rates in us-east-1. We believe the issue is caused by an unstable gossip node reporting inconsistent ring states to the cluster.

Given that we identified the issue was related to gossip and ring state inconsistencies, we are rolling out new gossip nodes across every region, which is rapidly resolving the issues.

We have stabilised the gossip and ring state globally now and error rates have reduced dramatically. There are few nodes that are still emitting channel errors which we are investigating.

Latencies and error rates are back to normal in all regions.

We sincerely apologise to customers who were affected by the incident, and will be posting a post-mortem once the investigation has completed.

We have completed the investigation of this incident and have written up a full post mortem at https://gist.github.com/paddybyers/3e215c0aa0aa143288e4dece6ec16285.

Any customers who have any questions or would like to discuss this incident should get in touch with the support and sales team at https://www.ably.io/support.

Once again, we sincerely apologise to customers who were affected by this incident. We are doing everything we can to learn from this incident and ensure that the service remains fully operational moving forwards.

Queues are now up and running normally. For a long time the partitioned state failed to heal, even after several rolling restarts. Ultimately we made the decision to stop and restart the entire cluster, which then took longer than expected due to an infrastructure issue.

We run our queue servers in 'pause minority' mode, which prioritises data integrity and preservation of messages already in the queue over availability to either consumers or new publishers for connections to the cluster in the minority.

We'll be investigating the causes of the partition, why it failed to heal, and the issues experienced in doing a stop-the-world restart, and post any updates here.

The us-east-1 region is continuing to experience a very high rate of errors, so we are now redirecting all traffic away from us-east-1 to us-west-1.

Error rates are quickly returning to normal. There are still some residual issues which we are resolving first before we investigate the root cause of the faults in us-east-1 that triggered the high error rates.

Error rates and availability metrics are now back to normal.

We are currently running with one fewer region than normal. Traffic that would have gone to us-east-1 is now going to us-west-1. We will updated this issue once we have identified the fault and brought the region back online.

Traffic is now being routed to us-east-1 again.

Please see the preliminary incident report on this incident: https://gist.github.com/paddybyers/3e215c0aa0aa143288e4dece6ec16285

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

Update at 14:03 (UTC): we are routing traffic away from us-east to other datacenters while we investigate the issue

Update at 14:30 (UTC): All regions are now affected by the present issues due to cascading failures. We are very sorry for the inconvenience and are trying to restore normal service as fast as possible.

Update at 16:03 (UTC): All regions are now stable and error rates have dropped dramatically. We will continue to inspect each region manually to ensure any issues are resolved forcibly.

Update at 17:30 (UTC): Error rates have been consistently back to normal in all regions. We believe all operations are back to normal. We will be completing a full post mortem now to understand what caused the global disruption, and importantly what caused the ripple effect from one region to another.

Update at 10:30 + 1 day (UTC): Our focus today is on addressing the immediate issues we identified yesterday that caused the significant increase in error rates and latencies globally. Whilst a post mortem is being prepared and will be published in due course, our priority at present is to resolve the underlying issue and prevent a repeat of the incident.

We are currently rolling out an update globally in all regions and expect this will take a few hours to complete.

We have published a summary of the incident affecting the Ably production cluster 30 October 2018, which includes our preliminary investigation and conclusions.

Please see https://gist.github.com/mattheworiordan/2abab206ee4e4859010da0375bcf4b1d for the report.

As mentioned in the report, we are sorry to all our customers for the disruption this resulted in, and are doing everything we can to ensure we are better equipped to deal with incidents more quickly and without disruption in future.

If you have any questions, please do get in touch with us at https://www.ably.io/contact