Here is the not-so-fun thing about running an email service provider: you get malicious actors trying to use your infrastructure for, well, malice — phishing, spoofing, et cetera.

We have a lot of defenses in place for this, but we detected someone with a relatively novel approach of trying to pass in problematic URLs which we weren't catching. Our other various systems did catch and apprehend this user before they were able to send any emails, but in our haste to push forward a solution ...

Post mortem

What broke?

workerscheduler, our process for running asynchronous jobs that are scheduled for some date in the future, was hard down for ~six hours. This meant, amongst other things:

1. Outbound emails were down
2. Cron was down
3. Other stuff, but those two dwarf everything else

Why did it break?

At a very high level, our asynchronous worker schedulers work something like this (none of this is bespoke, it's standard RQ):

1. To enqueue a job, serialize the method n...

We have identified the root cause of this issue, and a fix has been deployed.

This issue has been identified, and the fix has been shipped.

As written above, we've since recovered from the incident.