Previous incidents
Delays in background processing
Resolved Jan 31 at 03:52pm EST
Post mortem
What broke?
workerscheduler
, our process for running asynchronous jobs that are scheduled for some date in the future, was hard down for ~six hours. This meant, amongst other things:
- Outbound emails were down
- Cron was down
- Other stuff, but those two dwarf everything else
Why did it break?
At a very high level, our asynchronous worker schedulers work something like this (none of this is bespoke, it's standard RQ):
- To enqueue a job, serialize the method n...
3 previous updates
Unable to send from the author-facing app
Resolved Jan 27 at 01:52pm EST
We have identified the root cause of this issue, and a fix has been deployed.
1 previous update
Author-facing app failing to load
Resolved Jan 23 at 08:15pm EST
This issue has been identified, and the fix has been shipped.
1 previous update
Author-facing app failing to complete requests
Resolved Jan 17 at 09:26pm EST
As written above, we've since recovered from the incident.
1 previous update
Some tracked links may not be resolving
Resolved Nov 14 at 03:15pm EST
We're tracking reports of broken links for folks who have click tracking enabled. We've temporarily switched off click tracking for affected newsletters and are waiting to hear more from our upstream vendor.