All times are shown in UTC
This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in about 4 hoursThere was a minor outage in Europe Frankfurt for a few minutes during an upgrade. All other regions were operating correctly so traffic would have been routed to those regions.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in 2 minutesThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in about 4 hoursThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
There was a short loss of connectivity to this region with little or no disruption to realtime clients.
Resolved
in 6 minutesThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
The Heroku Postgres service was down for a brief period causing some downtime for the Ably website. We are looking at an upgrade now to help mitigate this type of issue in future
Resolved
in 7 minutesThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in about 15 hoursThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
An additional fault has been detected in realtime US West Oregon
15th Oct 11:31 PMAn additional fault has been detected in realtime Europe Frankfurt
15th Oct 11:31 PMAn additional fault has been detected in realtime South America
15th Oct 11:31 PMAn additional fault has been detected in realtime US West California
15th Oct 11:31 PMAn additional fault has been detected in realtime Australia
15th Oct 11:33 PMAn additional fault has been detected in realtime Europe Ireland
15th Oct 11:48 PMOur health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
The majority of regions were affected by a faulty deployment. During this time at least one region was up and running at any one time, so service continuity should have been maintained, albeit at reduced performance. We are currently integrating a change to our deployment process to prevent this happening again.
Resolved
in about 1 hourThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Some website features were effected by an incident in the realtime system related to a cluster state fault. We are investigating the root cause, but in the mean time, services have been resumed.
Resolved
in about 1 hourThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
An additional fault has been detected in realtime US West Oregon
13th Oct 01:11 PMAn additional fault has been detected in realtime US West California
13th Oct 01:12 PMAn additional fault has been detected in realtime US East Virginia
13th Oct 01:12 PMAn additional fault has been detected in realtime South America
13th Oct 01:12 PMAn additional fault has been detected in realtime Asia Singapore
13th Oct 01:15 PMAn additional fault has been detected in realtime Europe Frankfurt
13th Oct 01:17 PMAn additional fault has been detected in realtime Europe Ireland
13th Oct 01:59 PMWe are investigating the issue due to a faulty deployment.
13th Oct 02:39 PMWe have resolved the issue by forcibly removing all state from the cluster, and restarting the cluster. However, we are not sure of the root cause as yet, so will be focusing all of our energy on working out what went wrong today, and how we can prevent this in future. We'll update this ticket once we have a clearer idea of the problem.
Resolved
in about 2 hoursThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
The website was effected by a faulty deployment globally. Most services were operational throughout the effected period.
We are looking into ways to prevent issues like this in the future due to faults in our deployments.
Resolved
in about 2 hoursThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
A fault in all regions was detected
6th Oct 08:06 PMAn additional fault has been detected in realtime US West Oregon
6th Oct 09:07 PMOur deployment system had a fault and deployed an untested slug globally, resulting in some downtime. Globally, a lot of our services were operational throughout the effected period, and if using our client libraries, a failover region would have been used.
We are looking into ways to prevent issues like this in the future due to faults in our deployments.
Resolved
in about 1 hourThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in about 1 hourAn incident with Heroku is affecting our websites www.ably.io and status.ably.io.
See http://status.heroku.com/incidents/811 for further info.
Our realtime services are unaffected.
Heroku has reported that the issue is now resolved, and our own monitoring systems are reporting the websites being responsive as normal now.
Resolved
in about 1 hourThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in 1 minuteThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in 15 minutes