Ably Status

Incident log

14th August 2018 08:13:43 AM

High error rates during emergency revert of faulty deployment

At 5:50 UTC, a routine deploy caused error rates to climb in eu-central-1. See https://status.ably.io/incidents/553 for details.

Following an investigation into the root cause, we have decided that it is safer to revert the recent deployed on any instances globally until we can fully understand the root cause of the issue.

We have noticed error rates briefly increase in regions during (for periods of up to around 1 minute) whilst this code is being reverted.

We'll confirm once this process is completed.

14th Aug 09:00 AM

The new code has been reverted now on all nodes and error rates are back to normal.

We'll be completing a post mortem on this issue and post updates within this incident report.

Resolved

1 minute

14th August 2018 05:50:03 AM

eu-central-1 service disruption

Since 0550 UTC there was service disruption with increased error rates and latencies in eu-central-1. Recovery action is underway and service is getting back to normal.

More information will be posted here in due course.

14th Aug 07:42 AM

The root cause of this was due to a routine deploy in that region, which triggered an unexpected spike in CPU load across the routing layer for the entire eu-central-1 region. The error rates returned to normal at 6:41 UTC.

We are continuing to investigate why this deploy caused the disruption in this region.

14th Aug 09:15 AM

The code that caused this issue was rolled back in all regions across all instances where it had been deployed. See https://status.ably.io/incidents/554 for subsequent incident detailing the rollback process.

We will continue to investigate and publish a full post mortem in due course in incident https://status.ably.io/incidents/554.

Resolved

less than a minute

View archived incidents

Suspect a problem with the service?

Incident update emails

Subscribe to email notifications

Ably service health dashboard

Ably global service availability

In last 24hrs

In last 30 days

Data centre

Last 24hrs

Aug14

Aug13

Aug12

Aug11

Aug10

Aug09

Aug08

By data centre

Last 24hrs

Last 7 days

Aug14

Aug13

Aug12

Aug11

Aug10

Aug09

Aug08

Last 24hrs

Last 7 days

Aug14

Aug13

Aug12

Aug11

Aug10

Aug09

Aug08

Last 24hrs

Last 7 days

Aug14

Aug13

Aug12

Aug11

Aug10

Aug09

Aug08

Last 24hrs

Last 7 days

Aug14

Aug13

Aug12

Aug11

Aug10

Aug09

Aug08

Last 24hrs

Last 7 days

Aug14

Aug13

Aug12

Aug11

Aug10

Aug09

Aug08

Last 24hrs

Last 7 days

Aug14

Aug13

Aug12

Aug11

Aug10

Aug09

Aug08

Last 24hrs

Last 7 days

Aug14

Aug13

Aug12

Aug11

Aug10

Aug09

Aug08

Incident log

High error rates during emergency revert of faulty deployment

eu-central-1 service disruption