Back to overview

Emails were timing out

Mar 14 at 02:31pm EDT
Affected services
Backlog
p95 response time

Resolved
Mar 14 at 02:31pm EDT

Here is the not-so-fun thing about running an email service provider: you get malicious actors trying to use your infrastructure for, well, malice — phishing, spoofing, et cetera.

We have a lot of defenses in place for this, but we detected someone with a relatively novel approach of trying to pass in problematic URLs which we weren't catching. Our other various systems did catch and apprehend this user before they were able to send any emails, but in our haste to push forward a solution quickly we didn't backtest the performance of this solution — in particular, it added a very serious lag to particularly long emails, causing some of them to time out when you went to go and send them (either via the UI or the API.)

Normally, when something like this happens we just roll back to stem the bleeding; because this was also a bit of a security/fraud issue, we opted to roll forward and try and fix it live. We are all set now, but apologies if you had issues sending over the last 24 hours; we take that critical path seriously, and hope you understand (and forgive) the interruption.