Post mortem

What broke?

workerscheduler, our process for running asynchronous jobs that are scheduled for some date in the future, was hard down for ~six hours. This meant, amongst other things:

1. Outbound emails were down
2. Cron was down
3. Other stuff, but those two dwarf everything else

Why did it break?

At a very high level, our asynchronous worker schedulers work something like this (none of this is bespoke, it's standard RQ):

1. To enqueue a job, serialize the method n...

We have identified the root cause of this issue, and a fix has been deployed.

This issue has been identified, and the fix has been shipped.

As written above, we've since recovered from the incident.