www.ably.io
Back

Incident log archive

All times are shown in UTC

April 2016

9th April 2016 11:30:14 AM

Our automated health check system has reported an issue with website in website

This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.

9th Apr 12:32 PM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

Resolved

over 2 years ago
7th April 2016 04:17:34 AM

Our automated health check system has reported an issue with website in website

This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.

7th Apr 04:18 AM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

Resolved

over 2 years ago

March 2016

16th March 2016 02:34:26 AM

Our automated health check system has reported an issue with website in website

This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.

16th Mar 02:40 AM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

16th Mar 02:42 AM

The ably.io website experienced caching issues for 6 minutes, but the issue is now fixed.

Resolved

over 2 years ago

February 2016

24th February 2016 03:33:16 PM

Scheduled maintenance: 11pm UTC on 24 Feb 2016

We have scheduled maintenance to our global realtime infrastructure taking place at 11pm UTC time on the 24 Feb 2016.

Unfortunately we expect a few minutes of downtime globally as we are rolling out a change that requires all nodes to be upgraded simultaneously. This is due to us changing the communication protocol that all instances which is being done to help improve stability during network outages.

26th Feb 01:08 AM

This maintenance is now complete.

Closed

over 2 years ago
20th February 2016 01:15:05 PM

Our automated health check system has reported an issue with realtime cluster health in us-east-1-a

This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.

20th Feb 01:16 PM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

20th Feb 02:07 PM

For approximately 1 minute at 13:15, requests to one of our data centres were timing out. We are investigating the root cause, but the service has now returned to normal.

Resolved

over 2 years ago

January 2016

28th January 2016 07:01:55 AM

Degraded performance and intermittent failures

Following a routine recycle of our databases, our realtime system is attempting to connect to old database servers in all regions that do not exist.

This has resulted in an increased rate of 500, 401 and timeout errors.

We are looking into this now.

28th Jan 01:19 PM

We have identified the root cause of the issue and have fixed it.

We are also going to modify our monitoring systems to better detect intermittent failures as our monitoring systems were unfortunately reporting everything as healthy.

Resolved

almost 3 years ago
26th January 2016 11:00:32 AM

South America region taken offline due to AWS transient issues

At roughly 11:00am GMT, Amazon EC2 and ELB in the South America region have become unstable with transient intermittent issues.

We have taken the region offline and routed traffic to all other healthy data centres until Amazon resolves the issue.

26th Jan 01:17 PM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

26th Jan 01:18 PM

The issue has been resolved and all traffic is now being routed to South America again.

Update from Amazon:

"Between 2:15 AM and 4:33 AM PST we experienced connectivity issues for instances in a single Availability Zone and increased error rates for the EC2 APIs in the SA-EAST-1 Region. The issue has been resolved and the service is operating normally."

Resolved

almost 3 years ago
19th January 2016 08:17:17 PM

Our automated health check system detected a 1 minute issue in realtime Asia Singapore

This incident was created automatically by our automated health check system as it has identified a fault.

The issue appears to be a brief networking issue in that region.

19th Jan 08:18 PM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

Resolved

almost 3 years ago
15th January 2016 07:59:39 AM

Website tokens for demos and new sign ups failing

This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.

15th Jan 10:18 AM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

15th Jan 01:12 PM

Due to an upgrade to the realtime system some operations on the website were failing for a short period. The following website services were effected:
- Tokens used for demos and documentation examples
- New sign ups on the website

Resolved

almost 3 years ago
8th January 2016 11:02:43 PM

ably.io website caching issues

One of our Redis servers used for caching was unavailable for a period of 10 minutes resulting in intermittent issues with the website.

8th Jan 11:25 PM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

Resolved

almost 3 years ago
8th January 2016 09:11:32 AM

ably.io website cacheing issues

One of our Redis servers used for cacheing was unavailable for a period of 10 minutes resulting in intermittent issues with the website.

8th Jan 09:11 AM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

8th Jan 09:37 AM

The issue was resolved automatically by our health services

Resolved

almost 3 years ago

December 2015

15th December 2015 01:49:14 PM

Our automated health check system has reported an issue in realtime South America

This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.

15th Dec 01:50 PM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

Resolved

almost 3 years ago
15th December 2015 11:23:14 AM

Our automated health check system has reported an issue in realtime South America

This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.

15th Dec 11:25 AM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

Resolved

almost 3 years ago
12th December 2015 07:51:28 AM

Our automated health check system has reported an issue in realtime South America

This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.

12th Dec 07:52 AM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

Resolved

almost 3 years ago
11th December 2015 09:26:29 PM

Our automated health check system has reported an issue in realtime South America

This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.

11th Dec 09:27 PM

Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.

Resolved

almost 3 years ago