Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

There was a momentary loss of connectivity in the EU Central region. It lasted for approximately 1-2 minutes and resolved itself.

We identified the root cause of the issue was an internal monitoring system that went rogue and was generating significant load in the cluster.

We have disabled that specific monitoring system and will roll out a fix before turning that system back on.

All services are now back to normal.

At 18:40 the latencies returned to normal in us-east-1 following additional capacity being introduced into the cluster.

The issue was resolved by rolling back to a previous version of the realtime system. We will be trying to reproduce this issue in our lab environments now before rolling this out in production again.

The issue is now resolved.

We identified the root cause as memory starvation on an instance, however this in itself should have caused little or even no disruption whilst our health services recovered the problem.

The continued high error rates were unfortunately down to a problem in the Gossip protocol. We had indefinitely pending nodes that were not there i.e. they had been disposed of by the health services, yet the nodes in the cluster still reported the node as there. So the root cause of this issue is a Gossip fault which we are investigating now.

The traffic spike subsided within 5 minutes. We'll ensure there is a generous amount of extra capacity in place in this region for 24 hours whilst we investigate where this traffic originated.

We identified a number of nodes with memory exhaustion. We fixed the issue quickly but and are now investigating the root cause of the issue.

We have restored stability in Asia, however we are seeing gossip instability in other regions. We continue to investigate the root cause, but our priority remains to restore stability in other regions. We are recycling unstable nodes in other regions to resolve the issue.

The global realtime service stability has been resumed in full.

We are now investigating the root cause of the gossip instability to understand why a consensus was not reached soon after the abrupt exits in Asia. We'll also been investigating the cause of the abrupt exits we saw in Asia.

Once our investigation is complete, we'll be writing up a post mortem on the issues this evening.

Upon inspection of the logs at the time of the instability, it's clear we have a bug in our Gossip service that resulted in ghosts remaining in the ring that were in fact gone. We are investigating a few avenues which should lead to fixes soon.

The error rates have returned to normal. This was due to a new protocol deploy that should have been compatible, but unfortunately introduced unexpected communication issues.

As of 08:18 UTC, the issue has been resolved in all datacenters apart from eu-central-1. We are still investigating higher than normal error rates in eu-central-1.

The issues in eu-central are now resolved in full. We will provide a full update on the root cause an action we've taken soon.

We redeployed the instance which were unable to come to consensus on the ring. This resolved the issue.

We have however raised an internal ticket to now investigate why the ring was unable to reach stability within at most 10-15s, which is what we expect when a node is abruptly terminated.

It appears that the instability was caused by genuine network partitioning issues again.

We will continue to keep an eye on network issues.