Error rates have returned to normal.

We will continue to investigate the root cause of the increased error rates that lasted approximately 15 minutes.

We have completed a post mortem on what happened.

At the time of the issue, we deployed an update to the realtime platform that contained a regression relating to the progressive adoption of load as services come online during scaling operations. This meant that, for a short period of time early in the life of a new server, a small proportion of the channels handled by that server had errors. Externally, this meant that a small fraction of channels were inaccessible in us-east for a period of about 20 minutes.

The bug has since been fixed and we're looking into ways to add more simulated tests over the progressive load feature.

Error rates have returned to normal in all regions apart from eu-central-1.

We are investigating the errors in eu-central-1 now, although the error rate in that region is now very low.

As AWS issues in eu-west-1 and eu-central-1 continue (they have now confirmed networking issues), we are re-routing all traffic for the global cluster away from all EU regions. Please note that any traffic routed to the EU only (EU-only storage options, compliance reasons etc) unfortunately will continue to be routed to these clusters.

We believe the EU regions are now reaching stability and intend to route traffic back to EU in the next 15 minutes once some final testing is complete.

All global traffic is now being routed back to both EU datacenters, and everything appears to be normal. We'll continue to monitor closely now.

Whilst AWS is not yet reporting any issues on the status or personal health dashboards, we believe this issue is caused by a fault in the eu-central-1 and possibly eu-west-1 AWS regions. We are led to believe this as dedicated clusters, isolated from the global traffic in those regions, are also affected.

We will continue to investigate the issue and take action to minimise the impact.

AWS continues to report no issues, yet Twitter confirms the issues are widespread.

We have seen that eu-west-1 continues to exhibit problems, so we are now routing traffic away from eu-west-1 for now.

We are seeing stability return in eu-west-1 and eu-central-1. All eu-west-1 traffic is still being routed to other regions. Once eu-west-1 settles fully we'll redirect traffic back.

We are now investigating any residual issues caused by the partitions and instability to ensure no longer term impact on customer traffic.

Please note this issue is not resolved, but was mistakenly closed. We will continue in issue https://status.ably.io/incidents/562

Error rates have returned back to normal.

We will continue to investigate the root cause and update with a post mortem.

The error rates and latency have returned to normal (after 4 minutes). We believe this was caused by a deploy issue which we are investigating.

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

The upgrade has been rolled back whilst we investigate the root cause of the issue.

In total, the website was offline for some users for no more than 15 minutes.

The database migrations are now complete and all operations are back to normal.

The new code has been reverted now on all nodes and error rates are back to normal.

We'll be completing a post mortem on this issue and post updates within this incident report.

We have completed our post mortem on what happened:

The underlying cause was that we deployed code that was effectively broken. The specific issue was related to interoperability in a mixed cluster, and arose in a specific configuration that we had not tested sufficiently before release

When we deployed the code to eu-central-1, we saw a very large spike in system load - whose root cause was the bug - but due to the cascade of effects when hugely overloaded we were overwhelmed by logs containing secondary errors so we didn't recognise the root cause initially

The load then triggered autoscaling and recovery of nodes across the cluster which, due to a procedural error in making the deploy, caused other nodes to be restarted with the broken code.

We were able to recover the system by suspending recovery, finding the root cause bug, and reverting affected nodes.

However, moving forwards, we are going to improve our deployment strategy so that we can better test multi-region mixed cluster environments.

We apologize for the inconvenience this caused.

The root cause of this was due to a routine deploy in that region, which triggered an unexpected spike in CPU load across the routing layer for the entire eu-central-1 region. The error rates returned to normal at 6:41 UTC.

We are continuing to investigate why this deploy caused the disruption in this region.

The code that caused this issue was rolled back in all regions across all instances where it had been deployed. See https://status.ably.io/incidents/554 for subsequent incident detailing the rollback process.

We will continue to investigate and publish a full post mortem in due course in incident https://status.ably.io/incidents/554.

Until we understand the underlying issue fully, we are routing traffic away from the eu-west-1 datacenter as error rates are still high.

We are confident we have sufficient information now to understand and fix the root cause of the issue.

Our Cassandra upgrades will now be put on hold until a permanent fix is deployed to the global cluster.

We will shortly be directing traffic back to the eu-west-1 cluster. In the mean time, all traffic in EU is being handled by our datacenter in Germany.

All systems are back to normal in eu-west-1 Ireland as of 22:51.

We have a fix that we will deploy and test in the coming days, before we continue our scheduled Cassandra upgrades.

All traffic is now being routed back to eu-west-1 as normal.

Within 4 minutes the issue resolved itself and error rates have returned to normal.

It appears a routine Cassandra upgrade triggered this issue, however we are not yet clear on the root cause of the problem. We will continue to investigate.

Our health check system has reported this issue as resolved.
We will continue to investigate the iss0ue and will update this incident shortly.

We have identified an issue where some HTTP requests are intermittently stalled in the load balancer in the us-west-1 region.

The issue only affects us-west-1. As a precaution we are temporarily routing traffic away from that region. During this period clients that would have used that region will mostly now end up in us-east-1 (East Virginia). Aside from a possibly slight latency increase this should not cause any problems.

We were hoping that AWS would be able to resolve the issue in a sensible time so that they could diagnose the underlying problem on their end. However, as this did not happen time, we moved forwards and recycled the necessary load balancers and brought the US West region back online. All services are back to normal.

The maintenance is unfortunately taking longer than predicted as we are waiting on AWS to re-provision resources in that region. Once their re-provisioning is complete, the region will be back online.

The Sydney datacenter is now back online and operating normally.