We've identified that our hosting provider was incorrectly identifying incoming traffic as being from bots.

This issue is now resolved, and all services have been restored.

The incident is now resolved, and Heroku is back online. (HugOps to their team!)

All services are operational, and Buttondown emails are being sent without delay.

TL;DR

Bad configuration on one of our self-hosted SMTP servers caused a crash that proved difficult to recover from, leaving lots of emails “stuck” in varying degrees – and their being stuck manifested in a slew of unpleasant ways. We’ve fixed the configuration, are investing (literally, right at this very moment) in better tooling and alerting, and are architecting a way to prevent this from ever happening again.

The gory details

Buttondown uses a number of providers to actually send...

From 8:38pm EDT to 8:50pm EDT, we were serving 503s for around 75% of our incoming requests. This was purely due to a high burst of traffic that our scaling mitigated (albeit not quickly enough!) We're going to look into the problematic routes and harden their performance.

The change has been reverted, and subscribers have been restored to their correct status.

There is no action required from authors.

There was no impact to subscriber-facing features.