All times are shown in UTC
Following a spike we saw around 15 minutes ago, we raised an incident at https://status.ably.io/admin/incidents/550. Within 4 minutes the issue was resolved, and we marked the incident as resolved so that we could then focus on a post mortem. Unfortunately we are now experiencing a second spike in this region and will need to investigate things more thoroughly.
During this time nodes in eu-west-1 are rejecting some traffic causing traffic to be rerouted to other regions.
Until we understand the underlying issue fully, we are routing traffic away from the eu-west-1 datacenter as error rates are still high.31st Jul 10:49 PM
We are confident we have sufficient information now to understand and fix the root cause of the issue.
Our Cassandra upgrades will now be put on hold until a permanent fix is deployed to the global cluster.
We will shortly be directing traffic back to the eu-west-1 cluster. In the mean time, all traffic in EU is being handled by our datacenter in Germany.31st Jul 10:56 PM
All systems are back to normal in eu-west-1 Ireland as of 22:51.
We have a fix that we will deploy and test in the coming days, before we continue our scheduled Cassandra upgrades.
All traffic is now being routed back to eu-west-1 as normal.
Resolved3 months ago