TL;DR

Bad configuration on one of our self-hosted SMTP servers caused a crash that proved difficult to recover from, leaving lots of emails “stuck” in varying degrees – and their being stuck manifested in a slew of unpleasant ways. We’ve fixed the configuration, are investing (literally, right at this very moment) in better tooling and alerting, and are architecting a way to prevent this from ever happening again.

The gory details

Buttondown uses a number of providers to actually send...

From 8:38pm EDT to 8:50pm EDT, we were serving 503s for around 75% of our incoming requests. This was purely due to a high burst of traffic that our scaling mitigated (albeit not quickly enough!) We're going to look into the problematic routes and harden their performance.

The change has been reverted, and subscribers have been restored to their correct status.

There is no action required from authors.

There was no impact to subscriber-facing features.

We've processed the backlog.

We've confirmed and resent the majority of our affected emails, and are in touch with authors with whom we haven't automatically resent on their behalf.

Here is the not-so-fun thing about running an email service provider: you get malicious actors trying to use your infrastructure for, well, malice — phishing, spoofing, et cetera.

We have a lot of defenses in place for this, but we detected someone with a relatively novel approach of trying to pass in problematic URLs which we weren't catching. Our other various systems did catch and apprehend this user before they were able to send any emails, but in our haste to push forward a solution ...