Incident Details

All times are shown in UTC

31st July 2018 10:12:36 PM

eu-west-1 second spike

Following a spike we saw around 15 minutes ago, we raised an incident at https://status.ably.io/admin/incidents/550. Within 4 minutes the issue was resolved, and we marked the incident as resolved so that we could then focus on a post mortem. Unfortunately we are now experiencing a second spike in this region and will need to investigate things more thoroughly.

During this time nodes in eu-west-1 are rejecting some traffic causing traffic to be rerouted to other regions.

31st Jul 10:24 PM

Until we understand the underlying issue fully, we are routing traffic away from the eu-west-1 datacenter as error rates are still high.

31st Jul 10:49 PM

We are confident we have sufficient information now to understand and fix the root cause of the issue.

Our Cassandra upgrades will now be put on hold until a permanent fix is deployed to the global cluster.

We will shortly be directing traffic back to the eu-west-1 cluster. In the mean time, all traffic in EU is being handled by our datacenter in Germany.

31st Jul 10:56 PM

All systems are back to normal in eu-west-1 Ireland as of 22:51.

We have a fix that we will deploy and test in the coming days, before we continue our scheduled Cassandra upgrades.

All traffic is now being routed back to eu-west-1 as normal.


3 months ago