The issue is now resolved.

We identified the root cause as memory starvation on an instance, however this in itself should have caused little or even no disruption whilst our health services recovered the problem.

The continued high error rates were unfortunately down to a problem in the Gossip protocol. We had indefinitely pending nodes that were not there i.e. they had been disposed of by the health services, yet the nodes in the cluster still reported the node as there. So the root cause of this issue is a Gossip fault which we are investigating now.

The traffic spike subsided within 5 minutes. We'll ensure there is a generous amount of extra capacity in place in this region for 24 hours whilst we investigate where this traffic originated.

We identified a number of nodes with memory exhaustion. We fixed the issue quickly but and are now investigating the root cause of the issue.

We have restored stability in Asia, however we are seeing gossip instability in other regions. We continue to investigate the root cause, but our priority remains to restore stability in other regions. We are recycling unstable nodes in other regions to resolve the issue.

The global realtime service stability has been resumed in full.

We are now investigating the root cause of the gossip instability to understand why a consensus was not reached soon after the abrupt exits in Asia. We'll also been investigating the cause of the abrupt exits we saw in Asia.

Once our investigation is complete, we'll be writing up a post mortem on the issues this evening.

Upon inspection of the logs at the time of the instability, it's clear we have a bug in our Gossip service that resulted in ghosts remaining in the ring that were in fact gone. We are investigating a few avenues which should lead to fixes soon.

The error rates have returned to normal. This was due to a new protocol deploy that should have been compatible, but unfortunately introduced unexpected communication issues.

As of 08:18 UTC, the issue has been resolved in all datacenters apart from eu-central-1. We are still investigating higher than normal error rates in eu-central-1.

The issues in eu-central are now resolved in full. We will provide a full update on the root cause an action we've taken soon.

We redeployed the instance which were unable to come to consensus on the ring. This resolved the issue.

We have however raised an internal ticket to now investigate why the ring was unable to reach stability within at most 10-15s, which is what we expect when a node is abruptly terminated.

It appears that the instability was caused by genuine network partitioning issues again.

We will continue to keep an eye on network issues.

We are no longer seeing any network partitioning and the cluster is reported as stable in all regions.

Service is back to normal for almost everyone, except for some lingering issues limited to us-west-1

Everything is back to normal

POST MORTEM

The underlying cause of this issue was an upgrade to an upstream RPC library we use called gRPC (this is a Google maintained open source RPC library, see http://www.grpc.io/). Our strategy when upgrading upstream libraries is to roll forward to a stable release, then run this library in our CI , staging and sandbox environments for some time to monitor the impact. If everything is stable and our test suites are passing, we will then proceed to roll out to production.

In this instance, we followed normal procedure, rolled out to production, and everything at first appeared to be running fine. Unfortunately however, this was not the case and there was a native memory leak in gRPC v1.2 that caused the memory usage of all nodes to increase over time until eventually we had consistent failures globally across all nodes in the cluster at different times. Redeploying all instances resolved the issues, however the issue simply reoccurred later when the memory leak caused the instances to become starved of memory.

We did eventually isolate the root case as being that version of gRPC and reverted to version 1.1 and rolled this out globally.

WHAT WE COULD HAVE DONE BETTER

It is very difficult to balance the requirements of ensuring we keep our stack up to date and also minimise the impact of changes in libraries we are not maintaining ourselves. However, it's clear we need to have procedures in place to measure the impact of a change to an upstream library in a production environment. Unfortunately in our staging and sandbox environments, the load in those environments can hide faults like this which only then become obvious in production. Therefore, we will in future try to roll out upstream library changes in only one of our regions and compare that region with all others carefully over a substantial period of time before rolling out more widely. This should help to minimise the impact of a faulty upstream library in future.

UPDATE

Given the severity of the gRPC issue, we have now helped identify where the leak was introduced into the gRPC code base, have created a reproducible set of tests for the gRPC team to identify and fix the problem, and we have also got a PR waiting to land that will help to prevent upstream gRPC leaks going undetected. See https://github.com/grpc/grpc/issues/10445 for more info

Expanding incident to cover all regions. While the problem was in eu-west-1, customers in all regions may have been affected.

Error rates and response times are back to normal.

Please see https://status.ably.io/incidents/433 for post mortem on what the happened to cause this disruption, how we resolved the issue, and how we are looking to ensure changes like this don't have such wide impact on customers in future.

Amazon resolved the Route53 issues allowing updates to be made.

us-west-1 is operating normally now.

An additional fault has been detected in us-east-1-a

An additional fault has been detected in sa-east-1-a

An additional fault has been detected in us-west-1-a

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

An additional fault has been detected in us-west-2-a

The issue is now resolved, error rates are back to normal globally.