Previous incidents

The team has identified the issue and reverted the commit. The Dashboard is back up and working as expected.

Resolved.

We've addressed the root cause of the issue, and our systems are now fully operational.

We experienced a sudden increase in load that overwhelmed our previous process model, making it difficult for the system to recover gracefully once under stress.

To resolve this, we’ve shifted our load balancing approach and scaled up, which allows us to run multiple concurrent processes per instance. This not only improves overall capacity and responsiveness but also lets us restart individual processes...

What was happening

For the past few days, we’ve been experiencing thundering herd-esque downtime every time we deploy at the top of the hour. The top of the hour nuance is not actually that uncommon: we serve a lot of RSS traffic. (A fun fact is that 80% of our page views come from RSS readers and scrapers and the vast majority of them are fairly naive and run the equivalent of an hourly cron to ping every single RSS feed in which they are interested.)
After investigation, we were able t...

The issue is resolved, and archives are performing as expected.

We've identified that our hosting provider was incorrectly identifying incoming traffic as being from bots.

This issue is now resolved, and all services have been restored.

The incident is now resolved, and Heroku is back online. (HugOps to their team!)

All services are operational, and Buttondown emails are being sent without delay.