Previous incidents

January 2025
Jan 30, 2025
1 incident

Delays in background processing

Degraded

Resolved Jan 31 at 03:52pm EST

Post mortem

What broke?

workerscheduler, our process for running asynchronous jobs that are scheduled for some date in the future, was hard down for ~six hours. This meant, amongst other things:

  1. Outbound emails were down
  2. Cron was down
  3. Other stuff, but those two dwarf everything else

Why did it break?

At a very high level, our asynchronous worker schedulers work something like this (none of this is bespoke, it's standard RQ):

  1. To enqueue a job, serialize the method n...

3 previous updates

Jan 27, 2025
1 incident

Unable to send from the author-facing app

Degraded

Resolved Jan 27 at 01:52pm EST

We have identified the root cause of this issue, and a fix has been deployed.

1 previous update

Jan 23, 2025
1 incident

Author-facing app failing to load

Degraded

Resolved Jan 23 at 08:15pm EST

This issue has been identified, and the fix has been shipped.

1 previous update

Jan 17, 2025
1 incident

Author-facing app failing to complete requests

Degraded

Resolved Jan 17 at 09:26pm EST

As written above, we've since recovered from the incident.

1 previous update

December 2024
No incidents reported
November 2024
Nov 14, 2024
1 incident

Some tracked links may not be resolving

Resolved Nov 14 at 03:15pm EST

We're tracking reports of broken links for folks who have click tracking enabled. We've temporarily switched off click tracking for affected newsletters and are waiting to hear more from our upstream vendor.