www.ably.io
Back

Incident log archive

All times are shown in UTC

November 2015

13th November 2015 12:44:28 AM

Our automated health check system has reported an issue in realtime South America

This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.

13th Nov 12:46 AM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

13th Nov 12:56 AM

There was a short loss of connectivity to this region with little or no disruption to realtime clients.

Resolved

almost 3 years ago
5th November 2015 10:52:29 AM

Our automated health check system has reported an issue in website ably.io website

This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.

5th Nov 10:58 AM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

5th Nov 11:04 AM

The Heroku Postgres service was down for a brief period causing some downtime for the Ably website. We are looking at an upgrade now to help mitigate this type of issue in future

Resolved

almost 3 years ago

October 2015

28th October 2015 10:13:26 PM

Our automated health check system has reported an issue in realtime Asia Singapore

This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.

28th Oct 10:16 PM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

Resolved

about 3 years ago
15th October 2015 11:31:19 PM

Our automated health check system has reported an issue in realtime Asia Singapore

This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.

15th Oct 11:31 PM

An additional fault has been detected in realtime US West Oregon

15th Oct 11:31 PM

An additional fault has been detected in realtime Europe Frankfurt

15th Oct 11:31 PM

An additional fault has been detected in realtime South America

15th Oct 11:31 PM

An additional fault has been detected in realtime US West California

15th Oct 11:31 PM

An additional fault has been detected in realtime Australia

15th Oct 11:33 PM

An additional fault has been detected in realtime Europe Ireland

15th Oct 11:48 PM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

16th Oct 10:53 AM

The majority of regions were affected by a faulty deployment. During this time at least one region was up and running at any one time, so service continuity should have been maintained, albeit at reduced performance. We are currently integrating a change to our deployment process to prevent this happening again.

Resolved

about 3 years ago
13th October 2015 01:11:40 PM

Our automated health check system has reported an issue in website ably.io website

This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.

13th Oct 01:12 PM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

13th Oct 02:40 PM

Some website features were effected by an incident in the realtime system related to a cluster state fault. We are investigating the root cause, but in the mean time, services have been resumed.

Resolved

about 3 years ago
13th October 2015 12:47:47 PM

Our automated health check system has reported an issue in realtime Australia

This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.

13th Oct 12:49 PM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

13th Oct 01:07 PM

An additional fault has been detected in realtime US West Oregon

13th Oct 01:11 PM

An additional fault has been detected in realtime US West California

13th Oct 01:12 PM

An additional fault has been detected in realtime US East Virginia

13th Oct 01:12 PM

An additional fault has been detected in realtime South America

13th Oct 01:12 PM

An additional fault has been detected in realtime Asia Singapore

13th Oct 01:15 PM

An additional fault has been detected in realtime Europe Frankfurt

13th Oct 01:17 PM

An additional fault has been detected in realtime Europe Ireland

13th Oct 01:59 PM

We are investigating the issue due to a faulty deployment.

13th Oct 02:39 PM

We have resolved the issue by forcibly removing all state from the cluster, and restarting the cluster. However, we are not sure of the root cause as yet, so will be focusing all of our energy on working out what went wrong today, and how we can prevent this in future. We'll update this ticket once we have a clearer idea of the problem.

Resolved

about 3 years ago
6th October 2015 08:05:21 PM

Our automated health check system has reported an issue in website ably.io website

This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.

6th Oct 08:09 PM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

6th Oct 09:03 PM

The website was effected by a faulty deployment globally. Most services were operational throughout the effected period.
We are looking into ways to prevent issues like this in the future due to faults in our deployments.

Resolved

about 3 years ago
6th October 2015 07:47:42 PM

Our automated health check system has reported an issue in realtime South America

This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.

6th Oct 07:49 PM

A fault in all regions was detected

6th Oct 08:06 PM

An additional fault has been detected in realtime US West Oregon

6th Oct 09:07 PM

Our deployment system had a fault and deployed an untested slug globally, resulting in some downtime. Globally, a lot of our services were operational throughout the effected period, and if using our client libraries, a failover region would have been used.
We are looking into ways to prevent issues like this in the future due to faults in our deployments.

Resolved

about 3 years ago
1st October 2015 11:36:39 AM

Our automated health check system has reported an issue in website ably.io website

This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.

1st Oct 11:37 AM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

Resolved

about 3 years ago

September 2015

20th September 2015 02:57:56 PM

Heroku platform issues affecting reliability of our website platform

An incident with Heroku is affecting our websites www.ably.io and status.ably.io.

See http://status.heroku.com/incidents/811 for further info.

Our realtime services are unaffected.

20th Sep 02:58 PM

Heroku has reported that the issue is now resolved, and our own monitoring systems are reporting the websites being responsive as normal now.

Resolved

about 3 years ago

August 2015

18th August 2015 08:47:55 AM

Our automated health check system has reported an issue in realtime US West Oregon

This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.

18th Aug 08:48 AM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

Resolved

about 3 years ago

June 2015

30th June 2015 03:30:26 PM

Our automated health check system has reported an issue in realtime South America

This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.

30th Jun 03:30 PM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

Resolved

over 3 years ago
28th June 2015 01:16:00 PM

Performance issues during scheduled maintenance

This incident was created automatically by our automated health check system as it has identified a fault.

After reviewing the logs, during the recycle of all regions, some responses were too slow for our monitoring systems to see that as healthy. As such, this was reported as a fault.

The website is now operational again

28th Jun 01:17 PM

All data centres have been upgraded and the website is performing normally

Resolved

over 3 years ago
28th June 2015 12:24:23 PM

Scheduled maintenance

We are about to start scheduled maintenance on all regions.

There is no anticipated downtime, however each region will be taken off briefly and brought back up online. During this time, all traffic to that region will be re-routed to an alternative region.

28th Jun 12:48 PM

All data centres have been upgraded and a new region Frankfurt has been introduced.

Resolved

over 3 years ago
15th June 2015 09:28:51 PM

New data centers and regions come online - São Paulo, Australia and Singapore

We have introduced three new regions and six data centres to our cluster, São Paulo, Australia and Singapore.

This will dramatically reduce latencies in Asia and South America moving forwards and is available now to all customers.

Closed

over 3 years ago