Incident Details

All times are shown in UTC

28th February 2020 09:23:49 PM

Performance issues in all regions due to database layer issues

We are experiencing elevated error rates and latencies in all regions, due to continued intermittent performance issues we're experiencing with our database layer.

The issue has resolved within 5 minutes.

We're continuing our investigation into the root cause of these intermittent performance issues.

29th Feb 12:43 AM

Following more than 36 hours of intermittent short and significant increases in latencies in our persistence layer, the engineering team have been investigating the root cause to understand why only a small percentage of shards are affected during this time.

We have made significant progress in identifying potential root causes, however in the meantime we have also been addressing the issues by adding capacity and upgrading the entire cluster.

The persistence cluster is now operating with approximately 3x more capacity than it had 24 hours ago and is now upgraded to the latest stable versions.

We'll continue to investigate what triggered these increases in latencies, however we are optimistic that the increased capacity will now offer stability and predictable performance moving forward.

A full post mortem will be published soon.

10th Mar 08:25 PM

Please see https://status.ably.io/incidents/695 for the post mortem of this disruption.


in 8 minutes