www.ably.io

Ably service health dashboard

All times are shown in UTC

Ably global service availability

UP

In last 24hrs

NOW
AVAILABILITY

100.0%

DOWNTIME

0.0%

In last 30 days

TODAY
AVAILABILITY

100.0%

DOWNTIME

0.0%

Data centre

Last 24hrs

Aug14

Aug13

Aug12

Aug11

Aug10

Aug09

Aug08

By data centre

Asia Singapore

Last 24hrs

Last 7 days

Aug14

Aug13

Aug12

Aug11

Aug10

Aug09

Aug08

Australia

Last 24hrs

Last 7 days

Aug14

Aug13

Aug12

Aug11

Aug10

Aug09

Aug08

Europe Frankfurt

Last 24hrs

Last 7 days

Aug14

Aug13

Aug12

Aug11

Aug10

Aug09

Aug08

Europe Ireland

Last 24hrs

Last 7 days

Aug14

Aug13

Aug12

Aug11

Aug10

Aug09

Aug08

US East Virginia

Last 24hrs

Last 7 days

Aug14

Aug13

Aug12

Aug11

Aug10

Aug09

Aug08

US West California

Last 24hrs

Last 7 days

Aug14

Aug13

Aug12

Aug11

Aug10

Aug09

Aug08


ably.io website

Last 24hrs

Last 7 days

Aug14

Aug13

Aug12

Aug11

Aug10

Aug09

Aug08

Service operating normally

Performance issues

Service disruption

Loading...
View detailed API response times and roundtrip latency metrics

Incident log

14th August 2018 08:13:43 AM

High error rates during emergency revert of faulty deployment

At 5:50 UTC, a routine deploy caused error rates to climb in eu-central-1. See https://status.ably.io/incidents/553 for details.

Following an investigation into the root cause, we have decided that it is safer to revert the recent deployed on any instances globally until we can fully understand the root cause of the issue.

We have noticed error rates briefly increase in regions during (for periods of up to around 1 minute) whilst this code is being reverted.

We'll confirm once this process is completed.

14th Aug 09:00 AM

The new code has been reverted now on all nodes and error rates are back to normal.

We'll be completing a post mortem on this issue and post updates within this incident report.

Resolved

1 minute
14th August 2018 05:50:03 AM

eu-central-1 service disruption

Since 0550 UTC there was service disruption with increased error rates and latencies in eu-central-1. Recovery action is underway and service is getting back to normal.

More information will be posted here in due course.

14th Aug 07:42 AM

The root cause of this was due to a routine deploy in that region, which triggered an unexpected spike in CPU load across the routing layer for the entire eu-central-1 region. The error rates returned to normal at 6:41 UTC.

We are continuing to investigate why this deploy caused the disruption in this region.

14th Aug 09:15 AM

The code that caused this issue was rolled back in all regions across all instances where it had been deployed. See https://status.ably.io/incidents/554 for subsequent incident detailing the rollback process.

We will continue to investigate and publish a full post mortem in due course in incident https://status.ably.io/incidents/554.

Resolved

less than a minute
View archived incidents