All times are shown in UTC
At roughly 11:00am GMT, Amazon EC2 and ELB in the South America region have become unstable with transient intermittent issues.
We have taken the region offline and routed traffic to all other healthy data centres until Amazon resolves the issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
The issue has been resolved and all traffic is now being routed to South America again.
Update from Amazon:
"Between 2:15 AM and 4:33 AM PST we experienced connectivity issues for instances in a single Availability Zone and increased error rates for the EC2 APIs in the SA-EAST-1 Region. The issue has been resolved and the service is operating normally."
Resolved
in about 3 hoursThis incident was created automatically by our automated health check system as it has identified a fault.
The issue appears to be a brief networking issue in that region.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in 1 minuteThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Due to an upgrade to the realtime system some operations on the website were failing for a short period. The following website services were effected:
- Tokens used for demos and documentation examples
- New sign ups on the website
Resolved
in about 2 hoursOne of our Redis servers used for caching was unavailable for a period of 10 minutes resulting in intermittent issues with the website.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in 30 minutesOne of our Redis servers used for cacheing was unavailable for a period of 10 minutes resulting in intermittent issues with the website.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
The issue was resolved automatically by our health services
Resolved
in 9 minutesThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in 1 minuteThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in about 2 hoursThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in 1 minuteThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in about 4 hoursThere was a minor outage in Europe Frankfurt for a few minutes during an upgrade. All other regions were operating correctly so traffic would have been routed to those regions.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in 2 minutesThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in about 4 hoursThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
There was a short loss of connectivity to this region with little or no disruption to realtime clients.
Resolved
in 6 minutesThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
The Heroku Postgres service was down for a brief period causing some downtime for the Ably website. We are looking at an upgrade now to help mitigate this type of issue in future
Resolved
in 7 minutesThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
Our health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
Resolved
in about 15 hoursThis incident was created automatically by our automated health check system as it has identified a fault. We are now looking into this issue.
An additional fault has been detected in realtime US West Oregon
15th Oct 11:31 PMAn additional fault has been detected in realtime Europe Frankfurt
15th Oct 11:31 PMAn additional fault has been detected in realtime South America
15th Oct 11:31 PMAn additional fault has been detected in realtime US West California
15th Oct 11:31 PMAn additional fault has been detected in realtime Australia
15th Oct 11:33 PMAn additional fault has been detected in realtime Europe Ireland
15th Oct 11:48 PMOur health check system has reported this issue as resolved.
We will continue to investigate the issue and will update this incident shortly.
The majority of regions were affected by a faulty deployment. During this time at least one region was up and running at any one time, so service continuity should have been maintained, albeit at reduced performance. We are currently integrating a change to our deployment process to prevent this happening again.
Resolved
in about 1 hour