Incident Details

All times are shown in UTC

23rd April 2020 10:00:53 PM

Alert emails and other website notifications stalled

A website problem has been causing the sending of various automated emails, including limit notifications and welcome emails, to be stalled. The majority of emails arising from 0922 UTC on Thursday 23 April were stalled and backlogged until the service was unblocked at 1205 UTC on Monday 27 April. All backlogged emails were eventually sent by 1800 UTC.

The system is now operating normally.

An investigation is continuing into the circumstances that led to the problem, and to the extended time for resolution. This incident will be updated in due course with more information.

7th May 11:23 AM

Our engineering and ops teams have completed the post mortem of this incident and summarised all actions we have taken to ensure we can avoid any future disruption to our service.

See https://gist.github.com/paddybyers/c27d302524caa8e46f41e9ba19fdcf2e


in 4 days