All times are shown in UTC
This incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
The ably.io website experienced caching issues for 6 minutes, but the issue is now fixed.
Resolved
in 8 minutesWe have scheduled maintenance to our global realtime infrastructure taking place at 11pm UTC time on the 24 Feb 2016.
Unfortunately we expect a few minutes of downtime globally as we are rolling out a change that requires all nodes to be upgraded simultaneously. This is due to us changing the communication protocol that all instances which is being done to help improve stability during network outages.
This maintenance is now complete.
Closed
in 1 dayThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
For approximately 1 minute at 13:15, requests to one of our data centres were timing out. We are investigating the root cause, but the service has now returned to normal.
Resolved
in 1 minuteFollowing a routine recycle of our databases, our realtime system is attempting to connect to old database servers in all regions that do not exist.
This has resulted in an increased rate of 500, 401 and timeout errors.
We are looking into this now.
We have identified the root cause of the issue and have fixed it.
We are also going to modify our monitoring systems to better detect intermittent failures as our monitoring systems were unfortunately reporting everything as healthy.
Resolved
in about 5 hoursAt roughly 11:00am GMT, Amazon EC2 and ELB in the South America region have become unstable with transient intermittent issues.
We have taken the region offline and routed traffic to all other healthy data centres until Amazon resolves the issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
The issue has been resolved and all traffic is now being routed to South America again.
Update from Amazon:
"Between 2:15 AM and 4:33 AM PST we experienced connectivity issues for instances in a single Availability Zone and increased error rates for the EC2 APIs in the SA-EAST-1 Region. The issue has been resolved and the service is operating normally."
Resolved
in about 3 hoursThis incident was created automatically by our automated health check system as it has identified a fault.
The issue appears to be a brief networking issue in that region.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in 1 minuteThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Due to an upgrade to the realtime system some operations on the website were failing for a short period. The following website services were effected:
- Tokens used for demos and documentation examples
- New sign ups on the website
Resolved
in about 2 hoursOne of our Redis servers used for caching was unavailable for a period of 10 minutes resulting in intermittent issues with the website.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in 30 minutesOne of our Redis servers used for cacheing was unavailable for a period of 10 minutes resulting in intermittent issues with the website.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
The issue was resolved automatically by our health services
Resolved
in 9 minutesThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in 1 minuteThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in about 2 hoursThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in 1 minuteThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in about 4 hoursThere was a minor outage in Europe Frankfurt for a few minutes during an upgrade. All other regions were operating correctly so traffic would have been routed to those regions.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in 2 minutesThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in about 4 hours